AI QA Specialist

Logistics:

• Remote in CST
• Contract through EOY, with high likelihood of extensions and potential for perm conversion

Required Skills & Experience

5 years of experience testing AI and Machine Learning models

Proficient in both white box and black box testing methods.

Nice to Have Skills & Experience

Experience testing AI/ML solutions in AWS Experience integrating automated AI testing into CI/CD pipelines (e.g., GitHub Actions or similar tools).

Experience with monitoring and logging systems for post-deployment model validation.

Advanced experience testing generative AI systems, including LLMs for accuracy, bias, safety, and hallucinations.

Familiarity with RAG evaluation workflows and vector databases (e.g., FAISS, Pinecone, Weaviate). Experience with prompt engineering, adversarial prompting, and synthetic data generation.

Job Description

We are looking for AI Quality Assurance Specialists to join an Electric Utilities AI Delivery team in New Orleans, Louisiana to support the testing and validation of AI-powered applications, ML models, and generative AI solutions. This role blends traditional software QA skills with advanced evaluation methodologies for modern AI models, including LLMs and ML-driven product features. As a member of the team, you will work closely with experienced engineers and machine learning experts to qualify and refine features powered by vision, language, and cross-modal models. You will be responsible for designing rigorous evaluation strategies for both objective and subjective ML behaviors, creating reliable automated testing pipelines, and developing LLM-driven evaluators that complement human judgement.

Responsibilities include: · Create detailed, comprehensive, and well-structured test plans and test cases for traditional and ML-based solutions. Estimate, prioritize, plan, and coordinate testing activities. · Design, develop, and execute automation scripts. Identify, record, document, and track bugs. Perform thorough regression testing when bugs are resolved. · Develop and maintain robust testing frameworks for machine learning models · Develop and apply testing processes for new and existing products to meet client needs. · Monitor debugging process results. Investigate the causes of non-conforming software. Track quality assurance metrics, such as defect densities and open defect counts. · Collaborate with internal teams (e.g. developers, product managers, and designers) to identify testing requirements. · Identify and prioritize quality control issues from an end-to-end perspective. · Participate in scrum ceremonies such as refinement, demo, and release planning

Compensation:

$50/hr to $60/hr.

Exact compensation may vary based on several factors, including skills, experience, and education.

Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.