What are the responsibilities and job description for the Azure Databricks Engineer 10+ position at Jobs via Dice?
Dice is the leading career destination for tech experts at every stage of their careers. Our client, TekVivid, is seeking the following. Apply via Dice today!
Job Description: Senior Azure Databricks Engineer
Location Pleasanton, CA (Onsite)
EXP: 10 Years
About The Role
We are seeking a highly experienced Senior Azure Databricks Engineer to support digital transformation initiatives for our client. The ideal candidate has strong expertise in Azure cloud services, Databricks, Spark, large-scale data engineering, and end-to-end pipeline development. This is an onsite role requiring candidates to work from any of the client locations listed above.
Responsibilities
Job Description: Senior Azure Databricks Engineer
Location Pleasanton, CA (Onsite)
EXP: 10 Years
About The Role
We are seeking a highly experienced Senior Azure Databricks Engineer to support digital transformation initiatives for our client. The ideal candidate has strong expertise in Azure cloud services, Databricks, Spark, large-scale data engineering, and end-to-end pipeline development. This is an onsite role requiring candidates to work from any of the client locations listed above.
Responsibilities
- Design, build, and optimize end-to-end data pipelines using Azure Databricks and Apache Spark.
- Develop scalable ETL/ELT processes for structured and unstructured data.
- Integrate Databricks with various Azure services (ADLS, ADF, Azure SQL, Synapse, Key Vault, etc.).
- Implement Delta Lake features—time travel, ACID transactions, schema evolution, and optimization.
- Perform advanced data transformation, cleansing, aggregation, and feature engineering.
- Optimize Spark jobs for performance, reliability, and cost efficiency.
- Analyze and troubleshoot complex data issues in production environments.
- Collaborate with data architects, analysts, and business teams to define requirements & deliver solutions.
- Follow CI/CD best practices for data engineering using Git, Azure DevOps, or similar tools.
- Document solutions, create technical specifications, and support code reviews.
- 10years of professional Data Engineering experience.
- Strong hands-on expertise with Azure Databricks and Apache Spark (PySpark / Scala).
- Strong experience with Azure cloud services:
- Azure Data Lake (ADLS)
- Azure Data Factory (ADF)
- Azure SQL / Synapse
- Azure Key Vault
- Strong SQL development skills (complex queries, window functions, tuning).
- Experience with Delta Lake features and Databricks workspace management.
- Experience building and running pipelines on CI/CD frameworks (Azure DevOps preferred).
- Strong understanding of data modeling, partitioning, performance tuning, and job orchestration.
- Experience working with large datasets in distributed environments.
- Excellent communication and problem-solving skills.