Databricks DevOps Lead

Job Description(Databricks DevOps Lead- Spark configuration, unity catalog, terraform or IAAS Databricks administration (Remote only for candidate if they have Databricks Admin Certification) )

• Mandatory skills: Databricks, Spark, IAAS, Azure, Python, Cosmos DB, Azure DevOps (GitHub, CI/CD pipelines, Boards etc.), Docker & Azure Kubernetes Service, Graffana, JUnit, Postman, SonarQube
• Additional Skills: 5 years of DevOps/SRE experience supporting data platforms or distributed systems.
• Strong experience with CI/CD pipelines (Azure DevOps/GitHub Actions) for Databricks, Spark jobs, and Azure data components. Hands‑on experience with infrastructure as code (Terraform/Bicep/ARM) for provisioning Azure resources (Databricks, ADLS Gen2, Event Hubs, Key Vault). Solid understanding of build and release strategies for notebooks, JARs, wheels, and configuration artifacts. Experience setting up monitoring, logging, and alerting Spark jobs and data pipelines using Azure Monitor/Log Analytics or similar tools.
• Strong scripting skills (Python, PowerShell, or Bash) for automation and tooling.
• Good to have skills: Experience with containerization and orchestration (Docker, Kubernetes, AKS) for supporting ancillary services. Knowledge of security best practices (managed identities, secrets management, RBAC) for Azure data services. Exposure to reliability engineering concepts (error budgets, SLOs, SLI dashboards) for data workloads.

Roles & Responsibilities:

• Ability to create/maintain CI/CD pipelines (Azure DevOps/GitHub Actions) for Databricks, Spark jobs, and Azure data components.
• Develop infrastructure as code (Terraform/Bicep/ARM) for provisioning Azure resources (Databricks, ADLS Gen2, Event Hubs, Key Vault).
• Create scripts to manage release strategies for notebooks, JARs, wheels, and configuration artifacts.
• Set up monitoring, logging, and alerting Spark jobs and data pipelines using Azure Monitor/Log Analytics or similar tools
• Coordinate with offshore team in India