AI Agent Developer QA Specialization

AI Agent Developer QA Specialization

Location: Warren, NJ

Position type: W2 contract

Job Summary

We are seeking a QA domain expert with strong experience in AI development to design and build advanced QA Agents. This role is ideal for a QA engineer/developer who has transitioned to AI/ML, Large Language Models (LLMs), and agentic architectures. You will be responsible for architecting, implementing, and optimizing AI agents that automate, augment, and scale quality assurance processes within our engineering teams.

Key Responsibilities

• Analyze QA workflows to design agent architectures that automate requirement analysis, test generation, defect triage, and reporting.
• Build, fine-tune, and maintain QA agents using frameworks such as LangChain, OpenAI, LlamaIndex, or custom RL approaches.
• Engineer prompts and context strategies for LLM-powered test case generation, regression analysis, and intelligent reporting.
• Integrate agents with code repositories, CI/CD systems, test management platforms, and communication tools.
• Collaborate with QA, development, and AI/ML teams to collect requirements and continually improve agent performance.
• Implement agent validation, test coverage measurement, and reliability evaluation for agent-driven QA flows.
• Contribute to the continuous evolution of self-healing and adaptive QA agent features.
• Document agent design, workflows, evaluation metrics, and operational guidelines.
  
  

Required Skills & Qualifications

• Bachelor s or Master s in Computer Science, Engineering, or related field.
• 3 years in QA engineering, automation, or SDET roles.
• 1 years hands-on developing AI/LLM-based applications or agents (preferably for process automation or software quality).
• Strong programming skills: Python (must); knowledge of JavaScript/TypeScript, Java, or Go is an advantage.
• Proven experience with one or more of: LangChain, OpenAI API, LlamaIndex, Vertex AI, or similar agent and LLM frameworks.
• Experience integrating AI solutions with DevOps toolchains (Jenkins, GitHub, Jira, etc.).
• Deep understanding of software testing lifecycle, automation frameworks, and QA metrics.
• Clear communication skills with an ability to document and explain agent logic and technical rationale.
  
  

Bonus/Preferred Skills

• Experience designing agents for software quality test generation, autonomous triage, or regression analysis.
• Familiarity with RAG (Retrieval-Augmented Generation) and prompt tooling.
• Knowledge of agent frameworks (e.g., CrewAI, AutoGen, or similar).
• Contributions to open-source QA-AI tools or published research in autonomous QA.