What are the responsibilities and job description for the Azure Databricks Engineer / Lead position at HMG America?
HMG America LLC is the best Business Solutions focused Information Technology Company with IT consulting and services, software and web development, staff augmentation and other professional services. One of our direct clients is looking for Azure Databricks Engineer / Lead in Dallas, TX. Below is the detailed job description.
Role:Azure Databricks Engineer / Lead
Location: Dallas, TX Local candidates
Employment Type: Full-Time
Job Description
We are seeking an experienced Azure Databricks Engineer / Lead to join our team in Dallas, TX. The ideal candidate will have strong hands-on experience designing and building scalable data engineering solutions using Azure Databricks, PySpark, Python, SQL, and Azure Data Services.
Key Responsibilities
- Design, develop, and implement scalable data pipelines using Azure Databricks.
- Build and optimize ETL/ELT pipelines using PySpark, Python, and SQL.
- Develop data solutions using the Medallion Architecture (Bronze, Silver, and Gold layers).
- Work with Delta Lake, Delta tables, and advanced Databricks optimization techniques.
- Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault.
- Develop and manage Databricks Workflows, Jobs, and Notebooks.
- Implement data quality, monitoring, error handling, and performance optimization.
- Collaborate with Data Architects, Data Engineers, Data Scientists, and business stakeholders.
- Establish best practices for CI/CD, Git/version control, code reviews, and automated deployments.
- Lead technical discussions and provide guidance to junior engineers.
Required Skills
- 8 years of overall Data Engineering experience.
- Strong hands-on experience with Azure Databricks.
- Expert-level knowledge of PySpark and Spark SQL.
- Strong programming experience in Python and SQL.
- Experience with Delta Lake and Lakehouse architecture.
- Strong experience with Azure Data Lake Storage (ADLS Gen2).
- Experience with Azure Data Factory (ADF) and other Azure data services.
- Strong understanding of ETL/ELT, data modeling, and large-scale data pipelines.
- Experience with performance tuning and optimization in Databricks/Spark.
- Experience with Git, CI/CD, and DevOps practices.
- Excellent communication and stakeholder management skills.
Preferred Skills
- Databricks certifications.
- Experience with Unity Catalog and Databricks governance/security.
- Experience with Terraform or Infrastructure as Code.
- Knowledge of Azure DevOps.
- Experience working in a consulting or client-facing environment.