What are the responsibilities and job description for the Senior Databricks Engineer position at Golden Technology?
We are seeking a Senior Databricks Engineer with deep hands-on experience designing and implementing large-scale data solutions on Azure Databricks. The ideal candidate has real-world experience building and troubleshooting production-grade data pipelines, optimizing Spark workloads, managing Delta Lake architecture, and implementing DevOps best practices using IaC and CI/CD automation.
Key Responsibilities
- Design, develop, and maintain data pipelines and ETL solutions in Azure Databricks using PySpark and Delta Lake.
- Implement data integration frameworks and API-based ingestion using tools like Apigee or Kong.
- Analyze, design, and deliver enterprise data architecture solutions focusing on scalability, performance, and governance.
- Implement automation tools and CI/CD pipelines using Jenkins, Ansible, or Terraform.
- Troubleshoot production failures and performance bottlenecks — fix partitioning, caching, shuffle, cluster sizing, and Z-ordering issues.
- Manage Unity Catalog, enforce data security (row/column-level access), and maintain data lineage.
- Administer Databricks clusters, jobs, and SQL warehouses, optimizing costs through auto-stop, job clusters, and Photon usage.
- Collaborate with cross-functional teams to drive data strategy and standards across domains.
- Create and maintain detailed architectural diagrams, interface specs, and data flow documentation.
- Mentor junior engineers on Databricks, Spark optimization, and Azure data best practices.
Required Skills & Experience
- 5 years of experience as a Data Engineer with strong hands-on experience in Azure Databricks and PySpark.
- Solid understanding of Delta Lake, Z-ordering, partitioning, OPTIMIZE, and ACID transactions.