AI Quality Engineer / Lead

Job Description Sr. AI Quality Engineer / Lead

Experience: - Min 8 Years

Location: - Eden Prairie, MN & Chicago, IL - 3 days work from office

Duration: - C2H with client

Seeking an AI Quality Engineering / Lead professional to drive AI validation, LLM testing, runtime observability, automation, and reliability engineering for enterprise AI platforms, intelligent automation, and healthcare-focused AI applications.

Roles and Responsibilities: -

• Design, develop, and execute AI Quality Engineering strategies supporting AI-powered applications, large language model (LLM) solutions, intelligent automation, agentic systems, and enterprise AI platforms.
• Build and implement scalable AI Quality Engineering practices, including AI-native testing approaches, validation processes, runtime quality controls, reusable testing accelerators, and automated testing frameworks.
• Lead AI validation activities including functional testing, prompt validation, workflow testing, regression testing, release validation, runtime quality assurance, and production reliability support.
• Partner with AI Engineering, AIOps, LLMOps, Security, Governance, Clinical, Data, and Product teams to deliver scalable AI Quality Engineering processes across enterprise AI initiatives.
• Support runtime reliability through observability, telemetry, distributed tracing, monitoring, drift detection, incident response, and operational quality assurance for AI-enabled systems.
• Develop and maintain AI evaluation frameworks, validation datasets, quality scoring methodologies, and automated testing workflows that improve the reliability and scalability of AI solutions.
• Collaborate with Clinical, Operational, and Engineering stakeholders to validate healthcare workflows, payer operations, and AI-enabled business processes while supporting responsible AI deployment through human-in-the-loop validation practices.
• Coordinate testing activities across Agile delivery teams, including sprint planning, test execution, defect management, issue tracking, release readiness, risk identification, and production support.
• Mentor Quality Engineers and provide technical guidance that promotes engineering excellence, AI-enabled testing modernization, continuous improvement, and adoption of modern Quality Engineering practices.
• Research, evaluate, and recommend emerging AI Quality Engineering, testing automation, observability, and runtime assurance technologies to continuously improve enterprise AI.

Educational Qualifications: -

Engineering Degree – BE/ME/BTech/MTech/BSc/MSc.

Technical certification in multiple technologies is desirable.

Skills: -

Mandatory skills

• Experience in Quality Engineering, Quality Assurance, software testing, enterprise application delivery, technology operations, or related technology functions required.
• Min 8 or more years of experience providing technical leadership for testing initiatives, automation programs, or enterprise technology delivery projects required.
• Experience supporting Quality Engineering or Quality Assurance across enterprise platforms, APIs, healthcare applications, operational workflows, or integrated business systems required.
• Strong knowledge of software development life cycle (SDLC), Agile methodologies, test automation frameworks, defect management, release validation, and production support processes required.
• Experience validating AI-powered applications, intelligent automation, machine learning, large language model (LLM), or AI-enabled business workflows preferred.
• Experience with AI Quality Engineering practices, AI-assisted testing, runtime observability, monitoring, telemetry, or reliability engineering preferred.
• Strong analytical, problem-solving, organizational, communication, collaboration, and leadership skills required.
• Demonstrated ability to manage multiple priorities and deliver results within fast-paced, highly collaborative enterprise environments required.
• Experience in healthcare technology, payer operations, clinical workflows, or other regulated industries supporting AI governance and responsible AI deployment preferred.

Skills –

Quality LLM, OpenAI, AIOps, LLMOps, Quality Engineering AI Quality Engineering practices, AI-assisted testing, runtime observability, monitoring, telemetry, or reliability engineering.