At CI&T, we help large enterprises transform the potential of AI into real business impact with AI Deployment, AI-native execution, and tech-integrated business solutions.
With 30 years of experience in technological transformation, we accelerate innovation with expertise in Agentic SDLC, Application modernization, Data & AI, Martech and Business strategy.
We are 8,000 CI&Ters across more than 25 countries, collaborating to build solutions with real impact. AI is already part of how we work, evolve, and innovate every day.
Responsibilities:
- Build ingestion pipelines for Databricks using a medallion architecture (bronze, silver, and gold layers), leveraging Auto Loader and Delta Lake.
- Implement "before-and-after" reconciliation pipelines, including record counts, field-by-field comparisons, key/table mapping between versions, and large-scale validation rule application.
- Model the gold layer for reporting, covering approval/rejection rates, reasons for outcomes, and views segmented by cluster and object.
- Expose results via an API for a web application (using Databricks SQL Warehouse with the Statement Execution API, or a custom-built service) and document the API contract.
- Configure governance within Unity Catalog—including permissions, data lineage, and auditing—which are critical for client production data.
- Orchestrate processes using Workflows/Jobs, incorporate data quality checks (such as DLT expectations or similar), and manage version control via Databricks Asset Bundles.
Requirements:
- Experience in data engineering, with at least 2 years specifically using Databricks (PySpark, Delta Lake, Unity Catalog, Workflows).
- Advanced proficiency in SQL and Python.
- Experience with data reconciliation or validation within migration projects.
- Knowledge of SAP data structures.
- Experience in developing reports based on system data.
- Advanced English for conversation.