What are the responsibilities and job description for the Azure Databricks Lead Engineer position at Horizontal Talent?
We're seeking an experienced Azure Databricks Lead Engineer to help shape and manage a modern enterprise data platform. This role is ideal for a hands-on data engineer who enjoys building scalable solutions directly in Databricks, leading migration efforts, and driving automation across data workflows.
Responsibilities
Horizontal is committed to fostering an inclusive workplace where diverse perspectives are valued, respected, and encouraged. We welcome applicants from all backgrounds and are dedicated to creating an environment built on equity, belonging, and opportunity for everyone.
By applying for this position, you acknowledge and agree that Horizontal Talent may contact you regarding your application using automated technology, including phone calls, SMS/text messages, or email, which may be delivered by our virtual AI recruiter, Alex.
Responsibilities
- Design, build, and optimize end-to-end data pipelines within Azure Databricks and Delta Lake.
- Develop solutions using PySpark, SQL, or Scala to support core data transformations and platform engineering.
- Build and maintain production data workflows using Databricks Workflows and Jobs.
- Lead migration efforts from legacy data platforms into Databricks-based solutions.
- Refactor and convert existing business logic and ETL processes into scalable, code-driven Databricks notebooks.
- Implement data governance practices using Unity Catalog, including access, security, schema management, and lineage.
- Support CI/CD and DevOps automation for Databricks workspace deployments and jobs using Azure DevOps and YAML-based pipelines.
- Collaborate with stakeholders to ensure data solutions are reliable, maintainable, and aligned with business needs.
- Strong hands-on experience with Azure Databricks in an enterprise environment.
- Deep knowledge of Delta Lake and modern data engineering practices.
- Proficiency in PySpark, SQL, and/or Scala.
- Experience with Unity Catalog and Databricks governance capabilities.
- Working knowledge of CI/CD practices and DevOps automation.
- Experience building or supporting data platform migrations, especially from legacy systems.
- Ability to translate business and ETL logic into clear, maintainable code.
- Strong communication and collaboration skills.
- Understanding of Azure fundamentals.
- Databricks certifications, such as Data Engineer Associate or Professional.
- Microsoft Azure Data Engineer Associate (DP-203) certification.
- Microsoft Azure Fundamentals (AZ-900) or equivalent foundational Azure knowledge.
- Experience converting Alteryx workflows into PySpark or SQL solutions.
- Background supporting migration initiatives from Synapse into Databricks.
- Experience working in hybrid team environments and coordinating across multiple stakeholders.
Horizontal is committed to fostering an inclusive workplace where diverse perspectives are valued, respected, and encouraged. We welcome applicants from all backgrounds and are dedicated to creating an environment built on equity, belonging, and opportunity for everyone.
By applying for this position, you acknowledge and agree that Horizontal Talent may contact you regarding your application using automated technology, including phone calls, SMS/text messages, or email, which may be delivered by our virtual AI recruiter, Alex.