What are the responsibilities and job description for the AI Quality Engineer Lead position at Robustware?
Role: AI Quality Engineer Lead
Role: Onsite, Phoenix, AZ
This role will own the end-to-end quality strategy for AI Marketplace solutions, including QE agent development, AI validation, workflow orchestration, and platform integration. Define standards for AI testing, observability, security, and performance while enabling rapid adoption of reusable QE agents across projects. Provide technical leadership and mentorship to engineering teams building and deploying agentic solutions.
|
Competency Area |
Technical Expectations |
|
Agentic AI Architecture |
Understanding of AI agents, multi-agent workflows, MCP (Model Context Protocol), orchestration frameworks, tool calling, memory management, RAG patterns, and agent lifecycle management. |
|
AI/LLM Validation & Assurance |
Ability to define and implement testing strategies for AI systems including hallucination detection, response quality evaluation, guardrail testing, prompt validation, grounding verification, and reliability testing. |
|
QE Automation Engineering |
Strong hands-on experience with Playwright, Selenium, API automation, test frameworks, CI/CD integration, test data management, and automation architecture. |
|
Agent Development & Customization |
Ability to configure, extend, and customize agents for project-specific workflows, enterprise tools, APIs, business rules, and testing use cases. |
|
AI Observability & Monitoring |
Knowledge of agent telemetry, trace analysis, execution monitoring, prompt/response tracking, drift detection, performance analytics, and operational dashboards. |
|
Cloud & Platform Engineering |
Working knowledge of Azure AI, AWS Bedrock, OpenAI, Databricks, Kubernetes, containers, APIs, security integrations, and enterprise deployment patterns. |
|
Data & API Engineering |
Strong understanding of API contracts, service orchestration, structured/unstructured data, vector databases, embeddings, and data validation techniques. |
|
Responsible AI & Governance |
Experience validating security, privacy, compliance, explainability, bias detection, human-in-the-loop controls, and enterprise guardrails for AI agents. |
|
Performance & Scalability Testing |
Ability to validate agent response latency, concurrency, token consumption, workflow scalability, resiliency, and failover behavior. |
|
Solution Architecture & Consulting |
Capability to translate business use cases into agentic solutions, define reusable marketplace assets, establish standards, and mentor engineering teams. |
Ideal Skill Profile
Must Have
- Agentic AI / LLM fundamentals
- Playwright or modern automation framework
- API testing and integration
- AI testing and validation
- Azure AI/OpenAI ecosystem
- Strong QE architecture background
Good to Have
- LangChain / LangGraph
- AutoGen / CrewAI / Semantic Kernel
- Vector databases
- MCP-based architectures
- Kubernetes & containers
- Prompt engineering