What are the responsibilities and job description for the AWS Databricks Data Engineer position at Alltech Consulting Services, Inc.?
Job Title: Databricks Engineer (PySpark, AWS, Python)
Location: Plano, TX 5 Days onsite role(need local)
Overview
We are seeking a Data Engineer to lead the modernization of legacy ETL systems by migrating Ab Initio workflows to scalable, modular PySpark pipelines on Databricks. The role involves transforming complex data ecosystems into cloud-native architectures while ensuring data integrity, performance, and reliability.
Key Responsibilities
ETL Modernization & Development
Analyze and migrate legacy ETL workflows from Ab Initio to PySpark-based pipelines
Design and develop scalable data pipelines on Databricks
Refactor monolithic processes into modular, reusable components
Leverage existing enterprise datasets to avoid redundancy
Data Integration & Processing
Build and maintain ETL/ELT pipelines integrating data from Snowflake and other sources
Process and publish enriched datasets for downstream applications
Support batch and near real-time data processing
Data Lineage & Optimization
Create end-to-end data lineage and data flow diagrams
Identify redundancies and drive process consolidation and optimization
Ensure adherence to data governance and quality standards
Testing & Validation
Develop unit, integration, and reconciliation frameworks
Perform dual-run comparisons with legacy systems
Validate outputs in UAT and pre-production environments
Salary : $60 - $70