Demo

Databricks Engineer (PySpark and Data Lake)

Jobs via Dice
Wilmington, DE Full Time
POSTED ON 4/18/2026
AVAILABLE BEFORE 5/17/2026
Dice is the leading career destination for tech experts at every stage of their careers. Our client, HTD Resources, LLC, is seeking the following. Apply via Dice today!

Role: Databricks Engineer (PySpark and Data Lake)

Mode: Onsite

Duration: Long Term Contract

Location: Wilmington, DE (Local candidate will get the first preference)

Relocation: Yes

Job Description:

Overview

We are seeking a Data Engineer to lead the modernization of legacy ETL systems by migrating Ab Initio workflows to scalable, modular PySpark pipelines on Databricks. The role involves transforming complex data ecosystems into cloud-native architectures while ensuring data integrity, performance, and reliability.

Key Responsibilities

ETL Modernization & Development

  • Analyze and migrate legacy ETL workflows from Ab Initio to PySpark-based pipelines
  • Design and develop scalable data pipelines on Databricks
  • Refactor monolithic processes into modular, reusable components
  • Leverage existing enterprise datasets to avoid redundancy

Data Integration & Processing

  • Build and maintain ETL/ELT pipelines integrating data from Snowflake and other sources
  • Process and publish enriched datasets for downstream applications
  • Support batch and near real-time data processing

Data Lineage & Optimization

  • Create end-to-end data lineage and data flow diagrams
  • Identify redundancies and drive process consolidation and optimization
  • Ensure adherence to data governance and quality standards

Testing & Validation

  • Develop unit, integration, and reconciliation frameworks
  • Perform dual-run comparisons with legacy systems
  • Validate outputs in UAT and pre-production environments

Deployment & Operations

  • Support cutover and migration strategy from legacy systems
  • Decommission legacy workflows and optimize scheduling (e.g., Control-M)
  • Develop runbooks, monitoring, and operational documentation

Collaboration

  • Work with data architects, analysts, and downstream application teams
  • Coordinate user acceptance testing (UAT/FAT) and stakeholder sign-offs

Salary.com Estimation for Databricks Engineer (PySpark and Data Lake) in Wilmington, DE
$94,172 to $122,208
If your compensation planning software is too rigid to deploy winning incentive strategies, it’s time to find an adaptable solution. Compensation Planning
Enhance your organization's compensation strategy with salary data sets that HR and team managers can use to pay your staff right. Surveys & Data Sets

What is the career path for a Databricks Engineer (PySpark and Data Lake)?

Sign up to receive alerts about other jobs on the Databricks Engineer (PySpark and Data Lake) career path by checking the boxes next to the positions that interest you.
Income Estimation: 
$73,120 - $92,318
Income Estimation: 
$91,158 - $113,466
Income Estimation: 
$92,929 - $122,443
Income Estimation: 
$122,257 - $154,284
Employees: Get a Salary Increase
View Core, Job Family, and Industry Job Skills and Competency Data for more than 15,000 Job Titles Skills Library

Job openings at Jobs via Dice

Cable Technician ONSITE Cheyenne
Apply
  • Jobs via Dice Cheyenne, WY
  • Description The Field Service Technician will be required to perform both in-shop and field service work. Plans and coordinates activities concerned with i... more
  • Just Posted
Cable Technician - Data Center Technician
Apply
  • Jobs via Dice Cheyenne, WY
  • Data Center Technician - Cable Technician We are hiring Level 1-2 Data Center Technicians / Cable Technicians with a mix of inside plant (ISP) and outside ... more
  • Just Posted
Traveling Data Center Technician
Apply
  • Jobs via Dice Ellendale, ND
  • Description Maintain data center operations to ensure availability and reliability of service levels Conduct hardware and network diagnostics and repairs R... more
  • Just Posted
Beaker CP & DI Analysts
Apply
  • Jobs via Dice Sioux, SD
  • Description A local Sioux Falls healthcare system is looking to bring in two additional resources on their Laboratory Applications team -- a Beaker CP (Cli... more
  • Just Posted
View More Jobs at Jobs via Dice

Not the job you're looking for? Here are some other Databricks Engineer (PySpark and Data Lake) jobs in the Wilmington, DE area that may be a better fit.

Databricks Engineer
Apply
  • Envision Technology Solutions Wilmington, DE
  • Job Title: Databricks Engineer (PySpark and Data Lake) Location: Wilmington, DE – 5 Days onsite role Long Term Project Job Description We are seeking a Dat... more
  • Just Posted
Senior Databricks Engineer - Remote - USA
Apply
  • FullStack Dover, DE
  • About FullStack FullStack is one of the fastest-growing software consultancy companies in the Americas. We deliver transformational digital solutions to to... more
  • 7 Days Ago
View More Jobs

AI Assistant is available now!

Feel free to start your new journey!