AI/LLM Engineer

Senior SDET – AI / LLM Quality Engineering (Shared Services)

About The Team

This role sits within the QA Center of Excellence, as part of a small, highly specialized AI Quality Engineering team consisting of two SDETs and one Data Engineer.

The team operates as a shared service across the organization, defining how Large Language Model (LLM)–powered systems are tested, evaluated, observed, and trusted before and after production release.

Rather than building customer-facing AI features, this team builds LLM-based testing and evaluation frameworks and partners with product, platform, and data teams to ensure generative AI solutions meet quality, reliability, and compliance standards.

Role Overview

We are seeking a Senior Software Development Engineer in Test (SDET) with a strong automation and systems-testing background to focus on LLM quality, validation, and evaluation.

In This Role, You Will

• Test LLM-powered applications used across the enterprise
• Build LLM-driven testing and evaluation workflows
• Define organization-wide standards for GenAI quality and reliability
  
  

This is a hands-on engineering role with significant influence across teams.

Key Responsibilities

LLM Testing & Evaluation

• Design and implement test strategies for LLM-powered systems, including:
• Prompt and response validation
• Regression testing across model, prompt, and data changes
• Evaluation of accuracy, consistency, hallucinations, and safety
• Build and maintain LLM-based evaluation frameworks using tools such as DeepEval, MLflow, Langflow, and LangChain
• Develop synthetic and real-world test datasets in partnership with the Data Engineer
• Define quality thresholds, scoring mechanisms, and pass/fail criteria for GenAI systems
  

Test Automation & Framework Development

• Build and maintain automated test frameworks for:
• LLM APIs and services
• Agentic and RAG workflows
• Data and inference pipelines
• Integrate testing and evaluation into CI/CD pipelines, enforcing quality gates before production release
• Partner with engineering teams to improve testability and reliability of AI systems
• Perform root-cause analysis of failures related to model behavior, data quality, or orchestration logic
  

Observability & Monitoring

• Instrument LLM applications with Datadog LLM Observability to monitor:
• Latency, token usage, errors, and cost
• Quality regressions and performance anomalies
• Build dashboards and alerts focused on LLM quality, reliability, and drift
• Use production telemetry to continuously refine test coverage and evaluation strategies
  

Shared Services & Collaboration

• Act as a consultative partner to product, platform, and data teams adopting LLM technologies
• Provide guidance on:
• Test strategies for generative AI
• Prompt and workflow validation
• Release readiness and risk assessment
• Contribute to organization-wide standards and best practices for explaining, testing, and monitoring AI systems
• Participate in design and architecture reviews from a quality-first perspective
  

Engineering Excellence

• Advocate for automation-first testing, infrastructure as code, and continuous monitoring
• Drive adoption of Agile, DevOps, and CI/CD best practices within the AI quality space
• Conduct code reviews and promote secure, maintainable test frameworks
• Continuously improve internal tooling and frameworks used by the QA Center of Excellence
  
  

Required Skills & Experience

Core SDET Experience

• 5 years of experience in SDET, test automation, or quality engineering roles
• Strong Python development skills
• Experience testing backend systems, APIs, or distributed platforms
• Proven experience building and maintaining automation frameworks
• Comfort working with ambiguous, non-deterministic systems
  
  

AI / LLM Experience

• Hands-on experience testing or validating ML- or LLM-based systems
• Familiarity with LLM orchestration and evaluation tools such as:
• Langflow, LangChain
• DeepEval, MLflow
• Understanding of challenges unique to testing generative AI systems
  

Nice to Have

• Experience with Datadog (especially LLM Observability)
• Exposure to Hugging Face, PyTorch, or TensorFlow (usage-level)
• Experience testing RAG pipelines, VectorDBs, or data-driven platforms
• Background working in platform, shared services, or Center of Excellence teams
• Experience collaborating closely with data engineering or ML platform teams
  
  

What This Role Is Not

• ? Not a pure ML research or model training role
• ? Not a feature-focused backend engineering role
• ? Not manual QA
  
  

Why This Role Is Unique

• You will define how AI quality is measured across the organization
• You will build LLM-powered testing systems, not just test scripts
• You will influence multiple teams and products, not just one codebase
• You will work at the intersection of AI, automation, and reliability
  
  

#Remote