We are seeking highly skilled and experienced Azure Data Engineers to join a newly formed group concentrating on Data. Within this role you will be a key member of the team, working on a complex and challenging project within the Financial Services/Capital Markets industry. The primary focus of the role would be on building resilient, reusable Data Pipelines to extract, load, and transform raw data into a relational data model. The successful candidate will work across complex, multi-source datasets including loan servicing systems, property and valuation platforms, collections systems, and third-party data providers, delivering reliable and auditable data at scale.
- Build and maintain ingestion and transformation pipelines on Microsoft Fabric (Data Pipelines, PySpark notebooks, Dataflows Gen2) across the Landing, Curate and Enrich layers of the medallion architecture.
- Onboard new source systems by setting up the team’s reusable ingestion patterns (Fabric mirroring from Snowflake and SQL Server, change data capture, file and API loads). Do this through configuration rather than one-off code.
- Write and tune PySpark and T-SQL for Lakehouse and Warehouse workloads, including fact and dimension models in the enterprise data warehouse.
- Implement data quality rules and validation pipelines, and help investigate and fix data issues found by DQ checks and reconciliations.
- Work through the team’s engineering workflow: Git-based development in Azure DevOps, pull requests and peer review, and CI/CD promotion from Dev to UAT to Production.
- Contribute to the infrastructure-as-code (Terraform) and the orchestration config database (SQL projects / DACPAC) as needed.
- Work with Data Stewards, Data Analysts and the Data Architect to understand source data and turn mapping specs and business rules into working transformations.
- Keep documentation current, including source-to-target mappings, transformation rules and runbooks, so the work stays auditable and supportable.
- Support the platform day to day: monitor pipeline runs, triage failures and escalate where needed.
- Help prepare trusted datasets and Power BI semantic models for portfolio analytics, asset strategy, finance and risk.
Our tech stack- Platform: Microsoft Fabric (Lakehouse, Warehouse, Notebooks, Data Pipelines, Dataflows Gen2, Mirroring, OneLake)
- Languages: Python / PySpark, Spark SQL, T-SQL
- Sources: Snowflake, SQL Server, files and third-party feeds
- Azure: Azure SQL Database, ADLS Gen2, Key Vault, Data Factory, Entra ID (service principals and managed identities)
- Engineering: Azure DevOps (Repos, Pipelines, Boards), Git, fabric-cicd, Terraform / HCP Terraform, SQL Database Projects
- Governance and reporting: Microsoft Purview (data quality), Power BI (semantic models)
- Experience building data pipelines as a Data Engineer or in a similar role
- Strong Python/PySpark and SQL skills
- Hands-on experience with Microsoft Fabric, or with Databricks / Azure Synapse / Azure Data Factory and a willingness to move to Fabric
- Comfortable with Git, pull requests and CI/CD. Terraform or other infrastructure-as-code experience is a plus
- Understanding of data modelling (dimensional / medallion) and data quality practices
- Works well in a team, takes feedback in code review, and is keen to learn and develop
- Strong analytical, verbal and written communication skills
- Degree in Computer Science, Information Systems, Data Science or a related field preferred
- Experience in financial data domains (loan servicing, transactions, reference data, IBOR/ABOR) or in Real Estate, Credit, Banking or NPL Asset Management is desirable
- Microsoft certification (e.g. DP-700 Fabric Data Engineer) a plus