AI Quality & Automation Engineer (LLM Focused)

Location : Remote

• Communication could be stronger when explaining technical experience and solutions.
• Extensive of hands-on automation expertise
• Responses should be less focused on coordination more on technical implementation.
• Healthcare/insurance domain knowledge preferred
  
  

We build AI-powered solutions including conversational AI, RAG systems, AI agents, and enterprise automation platforms.

Role

Join the AI Engineering team to test and optimize Generative AI, conversational AI, and RAG applications. Focus on AI quality, benchmarking, safety, observability, and automation to ensure production-ready systems.

Responsibilities

• Test Generative AI and conversational systems
• Perform LLM evaluation and benchmark testing
• Validate RAG (retrieval quality, relevance, grounding)
• Build automation frameworks (Playwright/Selenium)
• Conduct API, regression, and E2E testing
• Perform AI safety, red teaming, and bias validation
• Support observability, monitoring, and HITL workflows
• Collaborate with engineering, QA, and DevOps teams
• Support CI/CD quality pipelines
  
  

Required Skills

• QA & automation testing
• Playwright/Selenium, API testing (Postman, REST)
• GenAI & RAG validation, LLM evaluation
• Prompt engineering, conversational AI testing
• HITL testing, AI safety & observability
• SQL, Git, Python, CI/CD concepts
  
  

Preferred

• OpenAI, Azure OpenAI, ChatGPT, Claude
• LangChain, CrewAI, LangGraph, MCP
• Vector DBs (Pinecone, ChromaDB, Weaviate)
• RAGAS, DeepEval
• Python/JS, Docker, Kubernetes, GitHub Actions