Director Data Engineering - QA Automation

- Core Specialty’s Data Environment is the organizational data ecosystem that supports enterprise requirements for data management, consolidation, insights, and reporting. The Data & Analytics organization ingests and manages data from over 200 distinct sources and growing, which serves business intelligence, AI/ML models, and enterprise decision-making. Maintaining the integrity, accuracy, and completeness of this extensive data environment requires a scalable and automated quality assurance strategy. The Director of Engineering – QA Automation will be responsible for designing and implementing this strategy, integrating quality processes throughout the data platform—including ingestion pipelines, semantic models, and machine learning workflows. This role will require 40% hands on involvement to start in order to build out the architecture, blueprints, and coding standards required for QA Engineers to adhere to. It may also include development efforts as needed. Key Accountabilities/Deliverables: Define and lead the QA automation strategy for data engineering, analytics, and machine learning initiatives. Architect and implement scalable, reusable automation frameworks for data pipelines, ingestion models, semantic layers, and ML workflows. Collaborate with Data Engineering, Data Science, and DevOps teams to integrate automated testing into CI/CD pipelines. Establish coding standards, architectural patterns, and best practices for test automation across the organization. Develop automated validation for data quality, schema enforcement, lineage tracking, and model performance. Mentor and grow a high-performing QA team with deep domain knowledge in data and analytics. Drive continuous improvement in testing tools, processes, and observability. In addition to the above key responsibilities, you may be required to undertake other duties from time to time as the Company may reasonably require. Technical Knowledge and Understanding: Programming Languages: Python (preferred), SQL, Java/Scala (for Spark-based pipelines) Automation Frameworks: PyTest, Great Expectations, dbt tests, Selenium, Airflow test suites Data Technologies: Spark, Kafka, Airflow, dbt, Delta Lake, Parquet, Avro Cloud Platforms: AWS (Glue, S3, Redshift), Azure, GCP (BigQuery, Dataflow), Databricks Data Warehouses: Snowflake, BigQuery, Redshift CI/CD & DevOps: GitHub Actions, Jenkins, Docker, Kubernetes ML Testing: MLflow, TensorFlow Model Analysis, custom validation pipelines Observability & Monitoring: Monte Carlo, Databand, OpenLineage SDLC: Agile/SCRUM Frameworks, Kanban, JIRA Data Quality Management: MS Purview Preferred Skills Experience with synthetic data generation and data fuzzing tools Experience with Data Contracts and the Producer vs. Consumer model Familiarity with data governance and compliance frameworks (e.g., GDPR, HIPAA) Exposure to AI-assisted testing platforms Experience: Bachelor’s degree in computer science, data engineering, or related field required Master’s degree in computer science, data engineering, or related field preferred Minimum of 15 years of experience in Data Engineering (hands-on roles) Minimum of 5 years of experience in a leadership position Prior QA and automation experience within data engineering space required Proven experience in testing large-scale data systems and ML workflows Strong architectural thinking and ability to define scalable testing patterns Excellent leadership, communication, and cross-functional collaboration skills Applicants must be authorized to work for any employer in the U.S. We are unable to sponsor or take over sponsorship of an employment Visa for this position. #LI-Hybrid - At Core Specialty, you will receive a competitive salary and opportunities for professional development and advancement. We offer medical, dental, vision, and life insurances; short and long-term disability; a Company-match of 100% of a 6% contribution 401(k) plan; an Employee Assistance Plan; Health Savings Account, Flexible Spending Account, Health Reimbursement Account, and a wellness program Core Specialty offers a diversified range of property and casualty insurance products for small to mid-sized businesses. From underwriting offices spanning the U.S., the Company focuses on niche markets, local distribution, and superior underwriting knowledge, offering traditional as well as innovative insurance solutions to meet the needs of its customers and brokers. For further information about Core Specialty, please visit www.corespecialtyinsurance.com. In compliance with the Transparency in Coverage rule, issued in 2020 by US Departments of Health, Human Services, Labor and Treasury our health plan’s machine-readable files (MRFs) can be accessed via www.Cigna.com.