What are the responsibilities and job description for the Data Architect (Azure + Databricks) position at Nine Core Technologies?
Data Architect (Azure Databricks) | Hybrid | Milpitas, CA
Location: Milpitas, CA (Hybrid – 4 Days Onsite) Employment Type: Contract
End Client: SanDisk (Through BayRock Labs)
We are hiring an experienced Data Architect with deep expertise in Azure, Databricks, and Apache Spark to join a high-impact enterprise data transformation initiative. This is a hands-on architecture role where you’ll be responsible for designing scalable data platforms while actively building, coding, and optimizing production-grade solutions.
If you’re an architect who enjoys coding or a senior data engineer ready to lead architecture, we’d love to hear from you.
Key Responsibilities
- Design scalable, secure, and cost-efficient enterprise data architectures on Azure and Databricks.
- Build and implement modern data lakehouse solutions using Delta Lake and Medallion Architecture.
- Develop production-grade data pipelines using PySpark, Scala, SQL, and Apache Spark.
- Optimize Databricks clusters, Spark workloads, and platform performance.
- Design streaming and batch processing solutions using Event Hubs, Kafka, and Structured Streaming.
- Implement data governance, security, and metadata management using Unity Catalog and Azure Purview.
- Conduct architecture reviews, code reviews, and mentor engineering teams.
- Collaborate with business and technical stakeholders to translate requirements into scalable solutions.
- Drive best practices around DevOps, Infrastructure as Code, CI/CD, testing, and observability.
Required Skills
- 8 years of Data Engineering/Data Architecture experience.
- Expert-level experience with Azure Data Services:
- Azure Data Lake Storage (ADLS)
- Azure Data Factory (ADF)
- Azure Synapse Analytics
- Azure Event Hubs
- Azure Purview
- Strong expertise in Databricks, including:
- Delta Lake
- Unity Catalog
- Medallion Architecture
- Cluster Optimization
- Advanced Apache Spark performance tuning and optimization.
- Strong coding experience in PySpark, Scala, Python, and advanced SQL.
- Experience designing enterprise-scale data platforms and modern data lakehouse architectures.
- Strong understanding of dimensional modeling, normalized models, and Data Vault.
- Experience with Infrastructure as Code (Terraform/ARM), CI/CD, and DevOps practices.
Preferred Qualifications
- Azure Solutions Architect Expert Certification.
- Databricks Certified Data Engineer Professional or Solutions Architect.
- Experience with MLOps and Machine Learning platforms.
- Cloud migration and modernization experience.
- FinOps and cloud cost optimization knowledge.
- Multi-cloud exposure (AWS/GCP) is a plus.
- Consulting or advisory experience.
What Makes This Role Unique
This is not a traditional architect position.
You’ll spend approximately:
- 70% Hands-on Development
- 30% Architecture & Technical Leadership
You’ll design enterprise architectures, write production code, optimize Spark jobs, troubleshoot production issues, and build working prototypes—leading through technical excellence.
Education
- Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related field.
- Master’s degree preferred.
Work Arrangement
- Hybrid – 4 Days Onsite
- Location: Milpitas, California
- Local Bay Area candidates only
- Face-to-Face Interview Required
If you’re passionate about building modern data platforms and want to work directly with a leading global technology client, we encourage you to apply.
Work Location: In person