What are the responsibilities and job description for the Databricks Developer position at Signature IT World Inc?
Role: Databricks Developer
Location: Boston, MA -Hybrid Onsite
Type: Contract
Job Summary
We are seeking a skilled Databricks Developer to design, develop, and optimize scalable data engineering solutions on the Databricks platform. The ideal candidate should have expertise in Apache Spark, Python, SQL, Delta Lake, and cloud platforms such as Azure, AWS, or GCP. The role involves building robust ETL/ELT pipelines, processing large-scale data, and collaborating with cross-functional teams to deliver high-quality data solutions.
Key Responsibilities
- Design, develop, and maintain data pipelines using Databricks and Apache Spark.
- Build scalable ETL/ELT workflows for batch and streaming data processing.
- Develop and optimize PySpark and Spark SQL applications.
- Implement data transformation and ingestion processes from multiple data sources.
- Work with Delta Lake to build reliable and performant data lakes.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Integrate Databricks solutions with cloud storage and cloud-native services.
- Collaborate with data architects, analysts, data scientists, and business stakeholders.
- Perform code reviews, testing, deployment, and production support.
- Monitor and troubleshoot data pipeline issues and performance bottlenecks.
- Follow CI/CD, Agile, and DevOps best practices.