What are the responsibilities and job description for the Databricks Developer position at DS Technologies Inc?
About US: We are a company that provides innovative, transformative IT services and solutions. We are passionate about helping our clients achieve their goals and exceed their expectations. We strive to provide the best possible experience for our clients and employees. We are committed to continuous improvement and innovation, and we are always looking for ways to improve our services and solutions. We believe in working collaboratively with our clients and employees to achieve success.
DS Technologies Inc is looking for Databricks Developer role for one of our premier clients.
Job Title: Databricks Developer
Location: Atlanta, GA (Hybrid)
Position Type: Contarct
Only W2
Position Type: Contarct
Only W2
Job Description:
A Pyspark and Databricks Developer with a good understanding of the entire ETL/Azure lifecycle with a background of data projects.
Responsibilities:
- Design, develop, and maintain scalable data pipelines and ETL processes using Azure Databricks, Data Factory, and other Azure services
- Implement and optimize Spark jobs, data transformations, and data processing workflows, Managing Databricks notebooks, Delta lake with Python, Delta Lake with Sparks SQL in Databricks
- Leverage Azure DevOps and CI/CD best practices to automate the deployment /DAB Deployments and management of data pipelines and infrastructure
- Ensure Data Integrity checks and Data Quality checks with zero percent errors when deployed to production
- Understand Databricks new features Unity Catalog/Lake flow/DAB Deployments/Catalog Federation
- Hands on experience Data extraction (extract, schemas, corrupt records, error handling, parallelized code), transformations and loads (user defined functions, join optimizations) and Production optimize (automate ETL)
Qualifications:
- Bachelor’s degree in computer science, Information Technology, or related field.
- Minimum of 5 years of experience in data engineering or similar roles.
- Proven expertise with Azure Databricks and data processing frameworks.
- Strong understanding of data warehousing, ETL processes, and data pipeline design.
- Experience with SQL, Python, and Spark.
- Excellent problem-solving and analytical skills.
- Effective communication and teamwork abilities.
Skills:
- Azure Databricks
- Python
- Apache Spark
- SQL
- ETL processes
- Data Warehousing
- Data Pipeline Design
- Cloud Architecture
- Performance Tuning
Flexible work from home options available.