Senior Azure Databricks Platform Engineer (Contractor)

Our financial services client is seeking a Senior Azure Databricks Platform Engineer (Contractor) to support and scale enterprise Azure Databricks platform services within a growing DevOps organization.

This role sits within the DevOps team responsible for designing, automating, and operating the Azure Databricks platform used by development and analytics teams across the organization.

The Contractor will work closely with engineering, infrastructure, and security teams to build and automate Azure Databricks environments, implement Infrastructure-as-Code, support data workloads, and provide platform services to internal customers.

This is a hands-on engineering role focused on Azure platform automation, Databricks infrastructure, and enabling scalable data processing workloads.

The ideal candidate combines deep experience in the Azure ecosystem, strong knowledge of Databricks platform architecture, and practical experience supporting Spark-based data processing environments.

Key Responsibilities

·         Design, deploy, and manage Azure Databricks workspaces

·         Configure and maintain Databricks clusters (interactive and job clusters)

·         Implement and manage cluster policies, permissions, and governance

·         Support hub/spoke network architecture for Databricks environments

·         Develop Infrastructure-as-Code using Bicep and ARM templates

·         Automate deployments for Databricks workspaces, networking, firewall rules, and permissions

·         Build and maintain CI/CD pipelines using Azure DevOps

·         Support PySpark and Spark-based data workloads

·         Enable Structured Streaming pipelines

·         Support Medallion architecture (Bronze, Silver, Gold)

·         Troubleshoot Spark SQL performance using query plans and execution plans

·         Support Azure Data Lake Storage, Azure Key Vault, and Azure Data Factory integrations

·         Troubleshoot workspace, cluster, and networking issues

·         Collaborate with Infrastructure, InfoSec, Data Engineering, and Development teams

Required Experience

·         5 years working with Microsoft Azure cloud environments

·         3 years working with Azure Databricks platforms

·         Hands-on experience managing Databricks workspaces, clusters, policies, and permissions

·         Strong experience with PySpark, Structured Streaming, and Spark SQL

·         Ability to analyze and optimize Spark query execution plans

·         Experience implementing Medallion architecture

·         Strong SQL proficiency including Spark SQL performance tuning

·         Experience building Infrastructure-as-Code using Bicep and ARM templates

·         Experience building CI/CD pipelines with Azure DevOps

·         Experience working with Azure Data Lake Storage, Azure Key Vault, and Azure Data Factory

·         Experience with Azure networking including VNETs, firewall rules, private endpoints, and hub/spoke architectures

·         Strong scripting skills in Python, Bash, or PowerShell

Nice-to-Haves

·         Azure Kubernetes

·         Apache Spark performance optimization

·         Enterprise data platform engineering

·         FinTech or regulated industry experience

·         Large-scale Azure platform environments