What are the responsibilities and job description for the Azure Databricks Administrator position at Bayforce?
Role Title: Azure Databricks Admin
Employment Type: Contract
Duration: 6 months to start, with chance of extension up to 18 months
Preferred Location: 4 days per week onsite in Wilmington, DE area
Role Description:
Urgent need for an Azure Databricks admin with 5 years experience and strong Terraform skills for ongoing support onsite in Wilmington. The position is for ongoing operational support, not a short-term project.
Requirements:
Minimum 4 years of hands on experience administering Azure Databricks platforms in an enterprise environment.
Proven experience configuring and managing Azure Databricks platform components, including:
- Clusters: autoscaling configurations, instance pools, cluster policies, and performance optimization
- Jobs and Workflows: scheduling, concurrency controls, retries, alerts, and notifications
- Workspace assets: notebooks, Git repos, libraries, init scripts, and secret scopes
- Strong experience with Unity Catalog and implementation of enterprise data governance models, including fine grained access controls and catalog level security.
- Experience with Databricks automation and deployment tooling, including:
- Databricks CLI
- Databricks REST APIs
- Databricks Asset Bundles (DAB)
- Experience integrating Databricks with BI Tools/analytics platforms.
- Solid understanding of data engineering patterns, Spark based workloads, and techniques for tuning performance and optimizing resource usage.
- Working knowledge of Microsoft Azure fundamentals, including:
- Identity and access management using RBAC, service principals, and managed identities
- Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Key Vault, Azure Monitor, and Log Analytics
- Azure networking concepts such as VNets, private endpoints, and DNS resolution
- Proven expertise with Terraform, including:
- Modular design patterns
- Remote state management
- Multi environment deployment strategies
- Secure infrastructure as code practices
- Strong experience building and maintaining CI/CD pipelines using GitLab CI/CD, including:
- YAML based pipeline definitions
- Runner configuration
- Environment promotion strategies
- Approval gates and controlled deployments
- Demonstrated experience supporting compliance driven and regulated environments, including:
- Audit evidence collection and documentation
- Periodic access reviews
- Change management and release controls
- Nice to Have experience on MLOps, including:
- Experience supporting machine learning workloads on Azure Databricks
- Familiarity with MLflow for experiment tracking, model registry, and lifecycle management
- Exposure to MLOps pipelines, including model training, validation, and deployment automation
- Understanding of model governance, versioning, and promotion across environments