This role is for one of Weekday’s clients
Salary range: Rs 2000000 - Rs 4000000 (ie INR 20 - 40 LPA)
Min Experience: 3+ years
Location: Bangalore
JobType: full-time
We are looking for a skilled and results-driven Senior Data Engineer to design, develop, and maintain scalable data pipelines and robust data infrastructure. The ideal candidate will have a strong understanding of data engineering principles, data architecture, ETL/ELT processes, and data integration techniques. This role involves transforming raw data into reliable, high-quality datasets that support business intelligence, analytics, reporting, and data-driven decision-making.
The candidate will collaborate with data scientists, analysts, software engineers, and business stakeholders to understand data requirements and deliver efficient, scalable, and secure data solutions. The role is suitable for professionals with 3–12 years of experience who are passionate about solving complex data challenges and improving data platform performance.
Requirements
Key Responsibilities
1. Data Pipeline Development and Management
- Design, develop, test, and maintain scalable data pipelines for batch and real-time data processing.
- Build efficient ETL/ELT workflows to extract, transform, and load data from multiple sources.
- Integrate structured and unstructured data from databases, APIs, cloud platforms, and external systems.
- Monitor pipeline performance, troubleshoot failures, and implement solutions to ensure reliability and availability.
- Automate recurring data processing tasks to improve operational efficiency.
2. Data Architecture and Infrastructure
- Develop and maintain scalable data architectures that support analytics and reporting requirements.
- Design and optimize data warehouses, data lakes, and lakehouse solutions.
- Work with relational and NoSQL databases to manage large volumes of data efficiently.
- Implement data partitioning, indexing, and storage optimization techniques to improve performance.
- Support cloud-based data infrastructure and contribute to platform scalability and cost optimization.
3. Data Quality and Governance
- Implement data validation, quality checks, and monitoring mechanisms to maintain data accuracy and consistency.
- Identify and resolve data discrepancies, duplication, and processing issues.
- Follow data security, privacy, access control, and governance standards.
- Maintain technical documentation for pipelines, data models, workflows, and system integrations.
4. Collaboration and Technical Excellence
- Collaborate with data analysts, data scientists, product teams, and business stakeholders to define data requirements.
- Translate business needs into technical data engineering solutions.
- Review code, follow engineering best practices, and contribute to continuous improvement.
- Identify opportunities to improve pipeline efficiency, reduce processing time, and optimize infrastructure costs.
- Stay updated on emerging technologies, tools, and industry practices in data engineering.
Required Qualifications
- 3–12 years of experience in data engineering, data platform development, or a related field.
- Strong programming skills in Python, SQL, or other relevant programming languages.
- Understanding of ETL/ELT processes, data modeling, and data warehousing concepts.
- Experience working with relational databases and large-scale data processing systems.
- Strong analytical, problem-solving, and troubleshooting skills.
- Ability to work independently and collaborate effectively with cross-functional teams.
Good-to-Have Skills
- Experience in Data Engineering, including pipeline design, data integration, and data transformation.
- Familiarity with Apache Spark, Kafka, Airflow, or similar data processing and orchestration tools.
- Exposure to cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
- Knowledge of Snowflake, Databricks, BigQuery, Redshift, or other modern data platforms.
- Experience with CI/CD, version control, workflow automation, and data observability tools.