What are the responsibilities and job description for the Sr. Databricks Engineer (PySpark and Data Lake) Exp. || 5 Days Onsite position at Jobs via Dice?
Dice is the leading career destination for tech experts at every stage of their careers. Our client, InfiCare Technologies, is seeking the following. Apply via Dice today!
Job Title: Sr. Databricks Engineer (PySpark and Data Lake) Exp.
Location: Wilmington, DE – 5 Days onsite role
Duration: Full Time / Long Term Project
Overview
We are seeking a Data Engineer to lead the modernization of legacy ETL systems by migrating Ab Initio workflows to scalable, modular PySpark pipelines on Databricks. The role involves transforming complex data ecosystems into cloud-native architectures while ensuring data integrity, performance, and reliability.
Key Responsibilities
ETL Modernization & Development
Job Title: Sr. Databricks Engineer (PySpark and Data Lake) Exp.
Location: Wilmington, DE – 5 Days onsite role
Duration: Full Time / Long Term Project
Overview
We are seeking a Data Engineer to lead the modernization of legacy ETL systems by migrating Ab Initio workflows to scalable, modular PySpark pipelines on Databricks. The role involves transforming complex data ecosystems into cloud-native architectures while ensuring data integrity, performance, and reliability.
Key Responsibilities
ETL Modernization & Development
- Analyze and migrate legacy ETL workflows from Ab Initio to PySpark-based pipelines
- Design and develop scalable data pipelines on Databricks
- Refactor monolithic processes into modular, reusable components
- Leverage existing enterprise datasets to avoid redundancy
- Build and maintain ETL/ELT pipelines integrating data from Snowflake and other sources
- Process and publish enriched datasets for downstream applications
- Support batch and near real-time data processing
- Create end-to-end data lineage and data flow diagrams
- Identify redundancies and drive process consolidation and optimization
- Ensure adherence to data governance and quality standards
- Develop unit, integration, and reconciliation frameworks
- Perform dual-run comparisons with legacy systems
- Validate outputs in UAT and pre-production environments
- Support cutover and migration strategy from legacy systems
- Decommission legacy workflows and optimize scheduling (e.g., Control-M)
- Develop runbooks, monitoring, and operational documentation
- Work with data architects, analysts, and downstream application teams
- Coordinate user acceptance testing (UAT/FAT) and stakeholder sign-offs