What are the responsibilities and job description for the Azure Databricks Data Engineer position at Jobs via Dice?
Dice is the leading career destination for tech experts at every stage of their careers. Our client, XFORIA Inc, is seeking the following. Apply via Dice today!
Role: Azure Databricks Data Engineer
Location: Minneapolis, MN – Onsite
Longterm contract
Job Description
Role: Azure Databricks Data Engineer
Location: Minneapolis, MN – Onsite
Longterm contract
Job Description
- 10–15 years of overall Data Engineering experience with 5 years of hands-on expertise in Azure Databricks, Azure Data Factory (ADF), and modern Azure data platforms.
- We are seeking a highly skilled Data Engineer with strong expertise in Azure Databricks, Azure Data Factory (ADF), and modern data architecture patterns including Medallion Architecture (Bronze-Silver-Gold) and Metadata-Driven Frameworks.
- The candidate will play a key role in designing, building, and optimizing scalable data pipelines and enterprise data platforms.
- Design and implement scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks
- Build and maintain data lake/lakehouse architectures following the Medallion Architecture (Bronze, Silver, Gold layers)
- Develop ETL/ELT pipelines for structured and semi-structured data
- Implement metadata-driven data ingestion frameworks to enable dynamic and reusable pipeline orchestration
- Optimize data processing using PySpark / Spark SQL in Databricks
- Ensure data quality, governance, and consistency across pipelines
- Work with business stakeholders to translate requirements into scalable technical solutions
- Implement data partitioning, performance tuning, and cost optimization strategies
- Automate workflows and monitor data pipelines for failures and performance issues
- Collaborate with architects and BI teams to support analytics and reporting layers
- Strong hands-on experience with:
- Azure Databricks (Spark, PySpark, Delta Lake)
- Azure Data Factory (ADF)
- Solid understanding of:
- Medallion Architecture (Bronze, Silver, Gold layers)
- Metadata-driven ingestion frameworks
- Experience in data modeling (star schema, dimensional modeling)
- Python / PySpark
- SQL (advanced querying and optimization)
- Experience with Azure Data Lake Storage (ADLS Gen2)
- Knowledge of Delta Lake concepts (ACID transactions, time travel)
- Understanding of batch and streaming data processing
- Hands-on experience designing reusable, scalable ETL frameworks
- Strong experience implementing config-driven / metadata-driven pipelines
- Familiarity with data governance, lineage, and cataloging tools