Lead Data QA

e&e is seeking a Lead Data QA for a hybrid contract opportunity in Philadelphia, PA!

The Lead Data QA is responsible for defining and driving the overall Data Quality Assurance strategy for enterprise-scale data platforms. This role ensures that all data systems meet rigorous standards for accuracy, performance, integration, security, and compliance. The Lead Data QA will provide leadership and mentorship to a team of data QA analysts and testers, establish quality frameworks for ETL/ELT pipelines, and integrate automation within Azure Data Factory (ADF), Databricks, and Snowflake environments. The ideal candidate possesses a deep understanding of data engineering, automation frameworks, and regulatory data compliance (HIPAA, CMS) within modern cloud architectures.

Responsibilities:

Leadership & Strategy

• Define and own the enterprise Data QA strategy encompassing functional, non-functional, integration, and performance testing.
• Lead and mentor a distributed team of Data QA professionals across multiple programs and data initiatives.
• Establish and maintain data quality SLAs, KPIs, and dashboards for critical datasets.
• Collaborate with data governance, engineering, and architecture teams to embed QA best practices across the data lifecycle.

Data Testing & Validation

• Design and implement automated test plans, scripts, and frameworks for ELT/ETL pipelines.
• Validate complex payer datasets including claims, membership, provider, and clinical data.
• Conduct FHIR-based API testing for CMS interoperability and compliance standards.
• Verify HEDIS measure calculations, healthcare quality metrics, and performance data accuracy.
• Log and track defects using appropriate QA tools; provide detailed feedback to engineering and architecture teams.

Automation Strategy & Framework

• Develop and implement a data QA automation framework for Databricks (Delta Live Tables, Delta constraints) and ADF pipelines.
• Utilize Great Expectations for reusable validation suites integrated into CI/CD workflows.
• Embed automated schema validation, reconciliation logic, and drift detection into data pipeline operations.

CI/CD Integration

• Develop QA gates and automated quality checks within Azure DevOps pipelines for Databricks Jobs/DLT, SQL metadata, and ADF deployments.
• Collaborate with DevOps and Engineering teams to embed QA automation into continuous integration and deployment processes.

Technical Delivery

• Partner with ADF, Databricks, and Snowflake teams to ensure end-to-end data quality.
• Build and maintain automation frameworks leveraging Python, PySpark, and SQL.
• Participate in code reviews, data model validation, and regression testing across environments.
• Work with business and data governance teams to identify, investigate, and remediate data quality issues.

Performance & Compliance

• Design and execute automated load and stress tests for large-scale pipelines and dataflows.
• Ensure all data QA processes align with HIPAA, CMS, and payer industry compliance standards.
• Support audits through proper documentation of QA processes, test results, and lineage verification.

Requirements:

Education:

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field.

Experience & Skills:

• 10 years of experience in Data QA/Testing, with at least 5 years in a leadership capacity.
• Strong proficiency with Azure Databricks (Delta Lake, Delta Live Tables, Unity Catalog).
• Hands-on experience with Azure Data Factory pipelines, monitoring, and CI/CD deployment.
• Advanced skills in Python, PySpark, and SQL for test automation.
• Experience with Great Expectations, Azure DevOps, and data quality automation frameworks.
• Familiarity with data governance, PII compliance, and enterprise data quality frameworks.
• Proven success integrating QA practices into DevOps pipelines within cloud data environments.
• Excellent communication, leadership, and cross-functional collaboration abilities.
• Experience in Agile/Scrum environments is a plus.

Preferred Qualifications:

• Experience with HL7/FHIR data models beyond payer use cases.
• Knowledge of Lakehouse and medallion architecture
• Familiarity with BI validation using Power BI or Tableau.
• Understanding of data governance platforms (e.g., Collibra).
• Prior experience designing data QA automation frameworks for pipelines and regression testing.
• Certifications such as Microsoft Certified: Azure Data Engineer Associate or Databricks Certified Data Engineer.