What are the responsibilities and job description for the Databricks Architect position at Capgemini?
Role: Databricks Architect
Location: Atlanta, GA / Chicago, IL / NY / NJ
Fulltime
We are looking for an experienced Databricks Architect to design and implement scalable data and machine learning platforms on Azure using Databricks. The ideal candidate will lead architecture decisions, ensure best practices in MLOps, and enable secure, automated workflows for data engineering and ML model deployment.
Key Responsibilities
- Define and implement end-to-end architecture for data and ML platforms using Databricks.
- Design data pipelines and ETL workflows leveraging Python, PySpark/Spark.
- Architect MLOps frameworks using MLFlow, Databricks Asset Bundles, and Feature Store.
- Establish CI/CD pipelines and DevSecOps practices using Azure DevOps.
- Optimize Databricks clusters, job orchestration, and cost management.
- Ensure security, compliance, and governance across all data and ML workflows.
- Collaborate with data engineers, data scientists, and cloud teams to deliver robust solutions.
Required Skills
- Databricks Expertise: Workspace architecture, Asset Bundles, Feature Store.
- Programming: Python, PySpark/Spark.
- MLOps: MLFlow, model lifecycle management.
- Cloud & DevOps: Azure, Azure DevOps, CI/CD, DevSecOps.
- Strong knowledge of distributed computing, data lakehouse architecture, and performance tuning.
