AI Engineer

Job Description

AI Agent Developer & Test Engineer

Overview

KēSTA I.T. is seeking an experienced AI Agent Developer & Test Engineer to join a cutting-edge team focused on building, testing, and supporting next-generation AI-powered applications. This role combines hands-on AI development with rigorous testing and quality assurance responsibilities, ensuring intelligent agent workflows perform reliably, securely, and consistently throughout the software development lifecycle.

Responsibilities

• Design, develop, and maintain AI agent workflows utilizing modern agent orchestration frameworks.
• Build and execute comprehensive testing strategies for AI agent systems, including unit, integration, regression, and end-to-end testing.
• Develop node-level test coverage to validate individual workflow components in isolation.
• Perform full workflow and graph-level testing to validate complete agent execution paths.
• Implement effective LLM mocking and simulation strategies to support deterministic and repeatable testing.
• Create and maintain regression test suites to ensure defects remain resolved over time.
• Design testing methodologies that address non-deterministic AI behaviors through controlled inputs, seeded testing, and assertion strategies.
• Develop and maintain automated end-to-end testing coverage for user-facing applications and workflows.
• Monitor test coverage metrics and continuously identify opportunities to improve system quality and reliability.
• Validate structured outputs, schema compliance, and data integrity across AI workflows.
• Implement production monitoring, logging, tracing, and observability practices for AI systems.
• Support cloud-based application deployments and ongoing operational maintenance.
• Collaborate closely with software engineers, architects, product owners, and QA teams to deliver high-quality AI solutions.
• Participate in troubleshooting, root-cause analysis, and production support activities.
  
  

Required Qualifications

• Professional experience developing AI-powered applications using agent frameworks such as LangGraph and/or LangChain.
• Strong Python development experience.
• Experience with automated testing frameworks including pytest and pytest-asyncio.
• Experience designing and executing unit tests for complex workflow-based applications.
• Experience testing distributed or orchestrated agent workflows from end to end.
• Strong understanding of test isolation, mocking strategies, and simulation techniques.
• Experience managing regression testing programs and quality assurance processes.
• Hands-on experience with Playwright or similar end-to-end testing frameworks.
• Experience producing and interpreting test coverage metrics and reporting.
• Familiarity with structured output validation and JSON schema testing.
• Experience implementing monitoring and observability solutions for production applications.
• Cloud application development and support experience within modern cloud platforms (GCP preferred).
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and collaboration abilities.
  
  

Preferred Qualifications

• Experience with AI evaluation and observability tools such as LangSmith, RAGAS, or similar platforms.
• Experience managing prompt versioning and prompt lifecycle processes.
• Knowledge of adversarial testing, red-team testing, and AI security validation methodologies.
• Experience monitoring token consumption, model utilization, and AI operational costs.
• Experience working with document extraction, OCR, or intelligent document processing solutions.
• Experience supporting applications within regulated industries such as financial services, healthcare, or government.
  
  

Preferred Platform & DevOps Experience

• Google Cloud Platform (GCP)
• Cloud Run
• Terraform
• Azure DevOps
• CI/CD pipeline design and automation
• Infrastructure-as-Code (IaC) best practices
  
  

Location

Midvale, UT

• On-site, Remote