What are the responsibilities and job description for the AI/LLM Automation Test Engineer position at Brilliant Infotech Inc.?
Role: Senior Quality Engineer – AI/LLM Test Automation
Location: Birmingham, AL - Hybrid
Type of Employment: Full Time Permanent
Job Description:
We are seeking a Senior Quality Engineer / SDET with strong hands-on experience in AI/LLM testing and test automation. The ideal candidate will have experience validating AI-powered applications and building LLM evaluation frameworks to assess accuracy, relevance, groundedness, hallucinations, reliability, and consistency of AI-generated responses.
Key Responsibilities:
- Build automated tests for AI/LLM applications using Playwright, TypeScript and/or Python.
- Develop LLM evaluation datasets, ground truth, regression suites, scoring criteria, and quality thresholds.
- Evaluate nondeterministic AI outputs using LLM-as-a-Judge, heuristic/code-based evaluators, and comparison-based evaluation.
- Test RAG, AI agents, prompts, tool/API calls, and multi-step workflows.
- Use LangSmith for tracing, datasets, experiments, and evaluations, and LangGraph or similar frameworks for agent workflow testing.
- Analyze AI failures across model behavior, prompts, retrieval, and application defects.
- Integrate AI evaluations and automated regression testing into CI/CD pipelines.
- Identify and resolve flaky tests and continuously improve test reliability.
Must-Have Skills:
- 10 years QA/SDET/Test Automation experience.
- 2–3 years hands-on AI/LLM testing and LLM evaluation.
- Experience with evaluation datasets, ground truth, LLM judges, scoring, and quality thresholds.
- Hands-on LangSmith and LangGraph/AI agent workflow experience.
- Strong Playwright with TypeScript and/or Python.
- API, integration, regression, and E2E testing experience.
- CI/CD, Git, and Agile development experience.
Salary : $120,000 - $130,000