Databricks Platform Engineer

Role: Databricks Platform Engineer (Azure DevOps)

Hybrid at Quincy, MA

Full Time Direct Hire Role

In-Person Interview is Mandatory

Job Description

Must Have Technical/Functional Skills

Key Responsibilities

Platform & Infrastructure

· Design, deploy, and manage Databricks workspaces across multiple Azure subscriptions and environments (dev, staging, production)

· Implement Infrastructure-as-Code (IaC) solutions using Terraform for Databricks resources

· Establish and maintain security best practices, including network isolation, RBAC, and data governance

· Configure and optimize cluster management, including auto-scaling policies and cost optimization strategies

· Manage Databricks workspace connectivity and networking (VNets, private endpoints, firewalls)

DevOps & Automation

· Build CI/CD pipelines using Azure DevOps for deploying Databricks notebooks, jobs, and configurations

· Implement automated testing and validation of data pipelines

· Create self-service deployment frameworks and templates for development teams

· Establish monitoring, alerting, and logging solutions for Databricks environments

· Automate workspace provisioning and deprovisioning processes

Collaboration & Enablement

· Partner with data engineers and analysts to understand platform requirements

· Provide technical guidance and documentation for platform usage

· Develop runbooks and operational procedures for incident response

· Mentor team members on Databricks best practices and Azure DevOps tools

Governance & Compliance

· Implement access controls and audit logging across Databricks workspaces

· Ensure compliance with organizational security and data governance policies

· Monitor and manage workspace usage, costs, and resource utilization

· Document architecture decisions and maintain platform documentation

Roles & Responsibilities

Required Qualifications

· 7 years of experience in platform engineering, infrastructure engineering, or DevOps

· 2 years of hands-on experience with Databricks infrastructure setup (or Apache Spark/Delta Lake environments)

· 3 years of experience with Azure cloud services and infrastructure

· 2 years of experience with Azure DevOps (Pipelines, Repos, Artifacts)

· Strong proficiency in Infrastructure-as-Code tools (Terraform Enterprise)

· Solid understanding of CI/CD principles and implementation

· Experience with Linux/Unix environments and scripting (Bash, Python, or similar)

· Familiarity with containerization technologies (Docker, Kubernetes)

Strong knowledge of networking, security, and identity management in cloud environments

Qualifications

BACHELOR OF COMPUTER SCIENCE

Thanks & Regards,

Vaishnav Gupta| Senior Technical Recruiter

KPG99, INC | www.kpgtech.com | MBE Certified Firm

3240 E State, St Ext | Hamilton, NJ 08619

Vaishnav.gupta@kpgtech.com