Data Engineering Lead
We are looking for a Data Engineering Lead with 8 to 12 years of experience to lead the build of data pipelines for an enterprise finance data platform on Azure, Databricks and Snowflake. You will run the data engineering team, own delivery from source systems through to certified Gold data products, and set the engineering standards the team works to.
KEY RESPONSIBILITIES
Lead a team of data engineers building ingestion and transformation pipelines
Deliver ingestion from finance source systems using change data capture, Azure Data Factory batch loads, API and event-driven feeds, and governed manual uploads
Build the Bronze, Silver and Gold layers in Databricks, including conformed dimensions and Gold data domains
Build Gold data domains for financial control, FP&A, tax, recharges and project accounting with the Data Architect
Publish certified data products from Databricks to Snowflake for reporting and analysis
Build data quality, reconciliation and matching checks into pipelines with the data quality team
Move recurring Dataiku analyst workflows into engineered Databricks pipelines
Set coding standards, branching, code review and CI/CD practices in Azure DevOps
Estimate work, plan sprints and report progress to the Engagement Manager
Tune pipeline performance and manage compute usage
Support production issues and handover to the support team
TECHNICAL SKILLS
Must-have
Databricks, including PySpark, Spark SQL, Delta Lake, Workflows and Unity Catalog
Snowflake SQL, data loading and performance tuning
Azure Data Factory
Python and advanced SQL
Dimensional modelling
Git and CI/CD with Azure DevOps
Leading engineering teams in agile delivery
Good-to-have
Informatica CDC and PowerCenter
API Management, Service Bus and Functions
Delta Live Tables
Dataiku
Microsoft Purview
DOMAIN KNOWLEDGE
Banking finance data, including general ledger, sub-ledgers, intra-group and recharges
Finance systems such as SAP ECC, BW and BPC, TM1, Alfa and other lending platforms
Treasury and banking data from systems such as TMS, T24 and bank statement feeds
CERTIFICATIONS (GOOD TO HAVE)
Databricks Certified Data Engineer Professional
SnowPro Advanced: Data Engineer
EXPERIENCE AND QUALIFICATIONS
8 to 12 years in data engineering, with at least 3 years leading teams
Delivered at least one cloud data platform on Databricks or Snowflake into production
Experience on data programmes in banking or financial services
Degree in engineering or computer a
Graduate/Post Graduate