What are the responsibilities and job description for the Lead Databricks Engineer with Python (Need Only local to MN) position at Sovereign Technologies?
Title: Senior / Lead Data Engineer
Location: Eagan, MN (Hybrid – 2 days/week onsite)
Duration: 3–6 Months Contract-to-Hire
Start Date: ASAP
Job Summary
We are seeking a Senior/Lead Data Engineer to modernize and own enterprise data pipelines by migrating legacy SAS-based workflows to Python/PySpark on Databricks. The engineer will partner with SMEs during the transition period and eventually take ownership of a critical healthcare analytics pipeline supporting HEDIS reporting.
Key Responsibilities
- Migrate legacy SAS pipelines to Python/PySpark on Databricks.
- Design, build, and maintain scalable ETL/ELT pipelines.
- Develop distributed data processing solutions using Databricks.
- Create and optimize complex SQL queries.
- Schedule, automate, and monitor data pipelines.
- Work with AWS services including S3, Lambda, Glue, and EC2.
- Manage Databricks notebooks, workflows, and clusters.
- Collaborate with business stakeholders and SMEs to transition pipeline ownership.
- Follow Agile methodologies and Git-based development practices.
Required Skills
- 8 years of Data Engineering experience.
- Strong Python programming and scripting skills.
- Hands-on PySpark development.
- Extensive Databricks experience (Notebooks, Workflows, Cluster Management).
- Experience processing large datasets in distributed environments.
- Strong SQL skills.
- AWS experience with:
- S3
- Glue
- Lambda
- EC2
- Experience building and automating ETL/ELT pipelines.
- Git/version control experience.
- Agile/Scrum experience.
Preferred Skills
- Experience with SAS-to-Python migration.
- AI/Automation experience.
- Healthcare or HEDIS domain experience.
- Lead/Principal Data Engineering experience.
Must-Have Skills
- Python
- PySpark
- Databricks
- SQL
- AWS (S3, Glue, Lambda, EC2)
- ETL/ELT Pipeline Development
- Distributed Data Processing
- Git
- Agile/Scrum
Nice-to-Have Skills
- SAS Modernization
- AI/Generative AI
- Healthcare/HEDIS
- Airflow or other workflow orchestration tools
- Leadership/Pipeline Ownership experience