Senior AI Agent & Evaluations Engineer

Job Description

Senior AI Agent & Evaluations Engineer

Company

Vacatia - Building the future of vacation ownership, transforming the industry through AI. Founded in 2013, headquartered in Mill Valley, USA, with 1001-5000 employees. Currently Late Stage.

Software

Remote

Job Summary

Design and improve AI agents that directly impact customer experiences and operational efficiency, while owning the intelligence layer behind these systems.

Responsibilities

• Design, refine, and optimize prompts, tool definitions, routing logic, and decision-making behavior across AI agent ecosystem
• Build and maintain evaluation frameworks, golden datasets, grading systems, and regression testing pipelines for agent quality and reliability
• Develop guardrails and safe-failure mechanisms for customer-facing and financially sensitive workflows
• Monitor production performance, investigate failures, identify edge cases, and continuously improve agent outcomes through data-driven iteration
• Partner with business stakeholders to translate policies and operational requirements into measurable agent behavior
• Collaborate with engineering teams on context requirements, tool contracts, and integration specifications
• Create scalable frameworks and reusable patterns for deploying AI agents across new workflows and use cases
• Establish best practices for prompt engineering, evaluation methodologies, observability, and agent operations
  
  

Qualifications

Required

• Proven experience shipping and owning production AI agents or LLM-powered systems beyond proof-of-concept
• Deep expertise in prompt engineering (system prompts, tool usage, context management, output constraints, agent behavior design)
• Hands-on experience building evaluation frameworks with golden datasets, scoring rubrics, LLM-as-judge, regression testing
• Strong familiarity with AI development tools like Claude Code, Codex, or similar coding agents
• Experience with agent observability/evaluation platforms (LangSmith, Langfuse, Arize, Galileo, or comparable)
• Ability to distinguish prompt issues from data, tooling, model, or evaluation failures
• Strong written and verbal communication skills across engineering and business teams
• Demonstrated ownership mindset with passion for building reliable, measurable, continuously improving AI systems
  
  

Preferred

• Experience with communication-based workflows (emails, support tickets, chat interactions, transcripts)
• Experience with multiple agent frameworks and practical understanding of their tradeoffs
• Production experience with online evaluation systems, automated scoring of live traffic
• Integration experience with Salesforce, AWS Connect, or customer engagement platforms
• Background in customer-facing industries with accuracy, compliance, and communication quality requirements
• Contributions to open-source projects, technical writing, or public thought leadership in AI, prompt engineering, or agent development