What are the responsibilities and job description for the Senior QA Engineer AI / LLM Testing position at Jobs via Dice?
Dice is the leading career destination for tech experts at every stage of their careers. Our client, HR Pundits, is seeking the following. Apply via Dice today!
Design and run intelligent test strategies for LLM and agent based systems, combining classic QA and test automation skills with a deep understanding of agentic automation, and conversational AI,
Responsibilities:
Design and run intelligent test strategies for LLM and agent based systems, combining classic QA and test automation skills with a deep understanding of agentic automation, and conversational AI,
Responsibilities:
- Design and execute end to end test plans for UX / Uber orchestrator/ Sub Agents . Build and maintain agentic AI test suites validating autonomous .
- Develop test data and synthetic scenarios for edge cases, ambiguous queries, and multilingual requests.
- Evaluate agent behavior through reasoning traces, tool use decisions, and memory usage.
- Implement and improve automated testing frameworks leveraging AI
- for conversational flows, APIs, and integrations.
- Collaborate with Product management / cross functional teams to refine requirements and acceptance criteria.
- Document defects, test results, and insights clearly for stakeholders.
- Minimum 5 years of experience in software QA / test engineering. Strong understanding of testing principles: functional, regression, integration, performance.
- Hands on experience testing AI or ML systems such as LLM chatbots or recommendation engines.
- Familiarity with key concepts of multi agentic systems, agentic frameworks (e.g., LangChain), AWS AgentCore.
- Scripting skills in Python and experience with REST API testing
- Hands on experience with AI and Machine Learning (ML) techniques to enhance test automation, such as generating test cases, predicting potential defects, or updating test code for user interface changes.
- Hands on experience with test automation frameworks and tools like Selenium, Playwright or Cucumber using scripting languages like Python, Java, or JavaScript and leveraging AI to improve productivity .
- Excellent analytical, documentation, and communication skills.
- Collaborate with cross functional teams for requirements clarifications and acceptance criteria.
- Document defects, test results, and insights clearly for stakeholders.