Databricks Platform Admin

Job Title _  Databricks Platform Admin

Location _ St. Louis, Missouri | 3 Days Onsite 2 Days Remote

Duration _12 Months Contract

Rate_ $ 50/hr

Job Description

What you'll do

·         Design, implement, and manage scalable and secure Databricks platforms to ensure optimal performance and reliability.

·         Create and manage Databricks workspaces and clusters, including right-sizing and lifecycle management of resources.

·         Automate infrastructure provisioning and deployment using Infrastructure-as-Code tools like Terraform, GitLab CI/CD, and scripting languages (Python, Bash, PowerShell).

·         Monitor platform health, performance, and cost by building dashboards and integrating with monitoring tools.

·         Troubleshoot and resolve issues related to cluster configuration, job failures, and performance bottlenecks within Databricks environments.

·         Collaborate with cross-functional teams including data scientists, ML engineers, and product teams to deploy and maintain data pipelines and services on Databricks.

·         Implement security and governance policies, including user management, access controls, and compliance adherence using Unity Catalog and other Databricks security features.

·         Optimize compute and storage resources for cost efficiency without compromising performance, leveraging features like Delta Lake and Photon.

·         Stay updated with new Databricks features and coordinate with product teams to evaluate and adopt them.

·         Document platform architecture, best practices, troubleshooting guides, and provide mentorship to junior engineers on platform usage and optimization

·         Being part of architect’s forum, interact with application owners and users and provide viable solutions based on their requirement

·         Coordinate between offshore team and customer stakeholders to ensure the requests are fulfilled on time

What you'll need

·         Required hands-on experience in architecting, deploying, managing, and supporting Azure Databricks Platform, Azure Datalake and related solutions.

·         Required hands-on experience working with Cloud provider tools and architecture (Azure) .

·         Required hands-on experience using scripting languages and command-line (CLI) tools such as Ansible, SQL, Terraform, Azure CLI, Databricks CLI, PowerShell and/or Bash.

·         Preferred experience in management and tuning of data platforms like Azure ADLS storage and Databricks.

·         Knowledge working with and maintaining CI/CD pipelines and versioning tools such as Git preferred.

·         Knowledge with a broad set of data ingestion and data pipeline patterns, and associated tools like (Qlik Replicate, Qlik Compose, Databricks, Kafka, Matillion ELT and Snowflake Services)

·         Experience with working in an agile product development team

·         Excellent communication skills and ability to articulate system designs and patterns to varying levels of leadership.

·         Own the solutions and demonstrate leadership in working with teams