What are the responsibilities and job description for the Azure Data Engineer position at Software Methods, Incorporated?
Software Methods is seeking an Azure Data Engineer, for a 6 month contract opportunity, to assist our client in their Chadds Ford, PA facility. They are requiring candidates to work three (3) days onsite.
If you’re the right one for us, you love working with data. You take great pride in your work, think of data transformation as more than just a job, and have a few great projects to show off. You can take a concept and transform it into an awesome data experience with your own technical and administrative expertise.
Responsibilities:
- Migrate data pipelines from Azure Data Factory to Databricks, rebuilding orchestration in Databricks Workflows (Lakeflow Jobs)
- Convert Synapse dedicated SQL pool objects, including T-SQL stored procedures, views, and tables, into Spark SQL, PySpark, and Delta Lake
- Redesign legacy data models into a Lakehouse Architecture (bronze, silver, and gold layers) rather than simply lifting and shifting them
- Validate migrated workloads through data reconciliation and parallel runs to confirm results match the legacy systems
- Build new pipelines in Databricks, using Delta Live Tables (Lakeflow Declarative Pipelines) where appropriate
- Set up and manage data governance and access controls in Unity Catalog
- Optimize Spark jobs, cluster configurations, and Databricks SQL warehouses for performance and cost
- Implement data quality checks, monitoring, and alerting across pipelines
- Deploy code through CI/CD using Git and Databricks Asset Bundles
- Document migration patterns, data models, and processes, and share knowledge with the wider team.
Required Experience:
- Bachelor’s degree or equivalent work experience
- Proven experience as a data engineer, with strong hands-on Databricks experience
- Proficiency in PySpark, Spark SQL, and Delta Lake
- Experience with Azure Data Factory and Azure Synapse dedicated SQL pools, including reading and converting existing T-SQL logic
- Hands-on experience migrating data workloads from a legacy platform to a modern Lakehouse
- Strong SQL skills and proficiency in Python
- Solid understanding of data modeling and Lakehouse Design
- Experience with Git and CI/CD workflows
- Strong communication skills and the ability to work both independently and collaboratively.
Preferred Experience:
- Databricks Certified Data Engineer (Associate or Professional) or Azure certifications such as DP-203
- Experience with Unity Catalog migrations or setting up governance from scratch
- Familiarity with Azure Data Lake Storage, Key Vault, and managed identities
- Experience with infrastructure as code (Terraform or Bicep)
- Experience supporting Power BI or other BI tools during a platform change.
If this sounds like you, please contact us today!
Unable to sponsor visas for this position.