What are the responsibilities and job description for the Databricks Engineer position at MphasiS Corporation USA?
Job Role: Databricks Developer
Location: Columbus, OH/ Wilmington, DE 5 Days onsite role
Long Term project, C2C is fine
Summary: We are looking for a Databricks Developer with strong experience in Python and Apache Spark to design, develop, and optimize scalable data pipelines and cloud-based data solutions. The ideal candidate will have hands-on expertise in Databricks, PySpark, SQL, and modern data engineering practices.
Key Responsibilities:-
Design, develop, and maintain ETL/ELT pipelines using Databricks and PySpark- Build scalable batch and real-time data processing solutions using Apache Spark- Develop data transformation workflows using Python, PySpark, and Spark SQL- Implement Delta Lake solutions and optimize data processing performance- Create and manage Databricks notebooks, jobs, and workflows- Ensure data quality, validation, logging, and exception handling- Optimize Spark performance using partitioning, caching, and tuning techniques- Collaborate with cross-functional teams to gather requirements and deliver solutions- Support CI/CD, deployment automation, and production releases- Troubleshoot and resolve issues in data pipelines and platform processes
Required Skills:-
Strong hands-on experience with Databricks
Strong Python programming skills
Expertise in Apache Spark, PySpark, and Spark SQL- Strong SQL and data transformation skills
Experience with Delta Lake and lakehouse concepts- Knowledge of ETL/ELT and data pipeline design
Experience with AWS, Azure, or Google Cloud Platform- Familiarity with Git, CI/CD, and Agile practices
Good problem-solving and communication skills
Salary : $100,000 - $140,000