Senior Data Engineer
We are looking for a Senior Data Engineer with 4 to 5 years of experience to build data pipelines for an enterprise finance data platform on Azure, Databricks and Snowflake. You will develop ingestion and transformation jobs, build Gold data products and publish them to Snowflake for reporting.
KEY RESPONSIBILITIES
Build ingestion pipelines from finance source systems using Azure Data Factory, change data capture, and API and event-driven feeds
Develop transformations in Databricks across the Bronze, Silver and Gold layers using PySpark and Spark SQL
Build conformed dimensions and fact tables to the agreed data model
Load certified Gold data into Snowflake and build the views used by reporting teams
Add data quality, reconciliation and logging checks to pipelines
Write unit tests and support system and user acceptance testing
Follow coding standards and take part in code reviews
Deploy code through Azure DevOps pipelines
Investigate and fix pipeline failures and data issues
Document pipelines, mappings and run procedures
TECHNICAL SKILLS
Must-have
Databricks, including PySpark, Spark SQL and Delta Lake
Azure Data Factory
Snowflake SQL and data loading
Python and advanced SQL
Dimensional modelling
Git and Azure DevOps
Good-to-have
Informatica CDC
Azure Functions and Service Bus
Unity Catalog
Delta Live Tables
Power BI
DOMAIN KNOWLEDGE
Finance data such as general ledger, trial balance, sub-ledgers and bank statements
Finance source systems such as SAP, TM1 and lending platforms
CERTIFICATIONS (GOOD TO HAVE)
Databricks Certified Data Engineer Associate
SnowPro Core
EXPERIENCE AND QUALIFICATIONS
4 to 5 years in data engineering, with at least 2 years on Databricks or Snowflake
Worked on at least one cloud data platform project that went into production
Degree in engineering or computer applications (B.E., B.Tech or MCA)
Degree in engineering or computer applications (B.E., B.Tech or MCA)