What are the responsibilities and job description for the Databricks Developer position at IT America?
Position: Databricks Developer
Location: Atlanta, GA / Nashville, TN
Duration: Long term contract
Note: Looking for Permanent / Visa independent consultants
We are looking for a skilled Databricks Developer to build and optimize data solutions on the Databricks platform, leveraging Azure cloud services to deliver scalable and high-performance data pipelines.
Key Responsibilities:
- Develop and optimize data pipelines and workflows using Databricks and PySpark
- Work with Delta Lake for reliable and performant data storage and processing
- Manage and govern data assets using Unity Catalog
- Write and optimize complex SQL queries for data transformation and analytics
- Collaborate with stakeholders to design and implement scalable data solutions
- Monitor and tune Databricks workloads for performance and cost efficiency
- Document solutions and contribute to engineering best practices
Required Skills & Qualifications:
- 4 years of experience in data engineering or a similar role
- Strong hands-on expertise with Databricks, including Delta Lake, Unity Catalog, and PySpark
- Advanced SQL skills, including query optimization and performance tuning
- Experience with Azure cloud services (e.g., Azure Data Factory, Azure Data Lake Storage, Azure Synapse)
- Strong understanding of performance optimization techniques within Databricks
- Knowledge of Medallion architecture
- Ability to work independently and collaborate effectively with cross-functional teams
Nice to Have:
- Experience with data migration projects
- Familiarity with CI/CD pipelines and DevOps practices
- Knowledge of other orchestration tools like Apache Airflow