This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Databricks Data Engineer and Scientist based in United States.
In this role, you will design, develop, and optimize enterprise-scale data pipelines within a Databricks and AWS-based Lakehouse environment. You will transform complex data into reliable, analytics-ready assets that support informed business decisions and advanced analytical initiatives. Working across data engineering, data modeling, and AI-assisted development, you will modernize legacy systems while ensuring data accuracy, performance, and scalability. You will contribute to a Bronze, Silver, and Gold medallion architecture and build robust ETL processes with comprehensive logging and error handling. The role combines hands-on technical delivery with collaboration across cross-functional teams in support of a critical federal data environment. This is a full-time, remote opportunity for an experienced professional who enjoys solving complex data challenges and improving enterprise data infrastructure.
Accountabilities
-
Design, build, deploy, and maintain end-to-end data pipelines for an enterprise Lakehouse using Databricks and AWS.
-
Develop and manage reliable ETL processes with comprehensive logging, error handling, troubleshooting, and issue resolution.
-
Design and maintain data models within a Bronze, Silver, and Gold medallion architecture, selecting appropriate dimensional or relational modeling approaches.
-
Analyze and decompose complex legacy SQL, rebuild pipelines and data models, and validate results for accuracy and performance.
-
Optimize data pipeline performance, scalability, and query execution using partitioning, bucketing, caching, and other relevant techniques.
-
Use AI-assisted coding tools to accelerate pipeline analysis, data engineering, debugging, and data modeling tasks.
-
Support data security and compliance requirements while collaborating with cross-functional teams to deliver high-quality, analytics-ready data.
-
Bachelor's degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, or Information Systems and Technology.
-
At least 8 years of experience designing, developing, deploying, and managing ETL processes and data pipelines.
-
At least 3 years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools, including Delta Lake and Unity Catalog.
-
At least 3 years of experience developing Python and PySpark code for ETL scripts and custom data transformations.
-
Practical experience using AI-assisted coding tools, such as Claude Code or Cursor, to support data engineering and data modeling activities.
-
Experience designing data models using dimensional and relational approaches within a Bronze, Silver, and Gold medallion architecture.
-
Hands-on experience with AWS services, including AWS Glue, Amazon S3, and Amazon RDS.
-
Knowledge of ETL and query performance optimization techniques, including partitioning, bucketing, and caching.
-
Demonstrated ability to analyze complex legacy SQL, reconstruct data pipelines and models, and compare outputs to validate accuracy and performance.
-
Ability to obtain and maintain a Public Trust or other required Suitability/Fitness determination, depending on client requirements.
-
U.S. citizenship is required to support the federal contract.
-
Work arrangement: Remote position based in the United States.
-
Employment type: Full-time.
-
Technical environment: Opportunity to work with Databricks, AWS, Lakehouse architecture, Python, PySpark, and AI-assisted development tools.
-
Compensation: Salary, bonuses, and other financial benefits were not specified in the provided job description and should be confirmed during the hiring process.
-
Additional benefits: Healthcare, paid time off, retirement plans, and other employee perks were not detailed in the provided description and should be confirmed with the hiring team.