AI Engineer – Agentic Systems & RLAIF

• We are building an RLAIF (Reinforcement Learning from AI Feedback) Plane: a governed learning system that turns production evidence from Client's customer-facing agents into better agent decisions—safely, measurably, and reversibly. Today, small errors compound across interconnected agents, missing capabilities and stale business context silently produce fluent-but-wrong answers, and prompt, policy, and model changes lack one governed path to independent proof.

• As an AI Engineer you will own the engineering behind that path: detecting capability and context failures before they become customer-visible, converting root-cause analysis into one-factor variants, running them through hard gates and calibrated AI judges, and promoting only what is approved, versioned, monitored, and reversible.

What You Will Do:

• Build the agent runtime- Design, build, and operate production agents and multi-agent workflows using LangGraph, LangChain, and deep-agent patterns—governed, monitored, and versioned from day one.

• Instrument for observability- Implement tracing, telemetry, and failure taxonomies across agents, tools, memory, and knowledge sources so every failure has a reproducible trail back to its root cause.

• Engineer the evaluation harness- Own golden sets, offline and online evals, and AI-judge pipelines with order-swap bias control and human calibration loops; report against business outcomes, not just model metrics.

• Drive harness optimization- Use observed failures to improve prompts, tools, skills, and rules through one-change-at-a-time experiments with regression-safe release: hard gates, approval, canary, and rollback.

• Separate the optimization paths- Keep harness, contextual-policy, and model-level changes distinct and independently evaluated; build the mechanisms that detect capability and knowledge-coverage gaps and propose governed updates.

• Ship on GCP- Deploy and scale the RLAIF Plane on Google Cloud (Vertex AI, GKE/Cloud Run, BigQuery, Pub/Sub) with CI/CD, IaC, and cost/latency controls.

• Collaborate and lead- Partner with architects, data scientists, SRE, and Client business stakeholders; document decisions, mentor engineers, and raise the bar on agent engineering practice.

Key Responsibilities:

• Ingesting and analyzing traces

• Reconstructing the sequence of events, tool calls, and business flow from traces

• Creating prompt/context/routing variants

• Defining and implementing evaluation dimensions

• Building and calibrating AI judges

• Designing scoring and reward-signal frameworks

• Generating autonomous optimization recommendations

Identifying root causes such as:

• Hallucinations

• Context drift

• Poor prompts

• Misrouting

• Policy/compliance issue

• Safety: Harmlessness, different biases, toxicity

• Missing or underperforming tools

• Taxonomy/ontology gaps

• Re-configuration opportunities (agent) for identification and recommendation and project outcomes.

What You Bring (Required):

• Demonstrable experience building and running AI agents in production—shipped systems serving real users, with ownership of reliability, latency, cost, and incident response.

• Hands-on experience with AI observability and evals: tracing frameworks (e.g., LangSmith, Arize Phoenix, OpenTelemetry), LLM-as-judge design, golden-set curation, and regression testing.

• Deep, practical experience with LangGraph and LangChain, including deep-agent patterns (planning, sub-agents, tool use, memory, human-in-the-loop, and state persistence).

• Strong Python engineering skills; comfortable with async services, APIs, containers, and testing discipline in a CI/CD environment.

• Working understanding of RLAIF/RLHF concepts, prompt and policy optimization, and how to isolate a single causal change in a complex system.

• 5 years of software engineering experience, including 2 years focused on LLM or agentic applications.

Preferred Qualifications:

• Cloud experience on Google Cloud (Vertex AI, GKE, Cloud Run, BigQuery, Pub/Sub, IAM); GCP certification is a plus.

• Experience with retrieval, enterprise knowledge bases, and memory systems with provenance, trust scoring, and decay.

• Familiarity with governed release practices in regulated or large-enterprise settings (approval workflows, canary, rollback, audit trails).

• Telecom or large-scale customer-service domain exposure; experience with MLOps tooling and feature flags.

The pay range that the employer in good faith reasonably expects to pay for this position is $39.30/hour - $61.40/hour. Our benefits include medical, dental, vision and retirement benefits. Applications will be accepted on an ongoing basis.

Tundra Technical Solutions is among North America’s leading providers of Staffing and Consulting Services. Our success and our clients’ success are built on a foundation of service excellence. We are an equal opportunity employer, and we do not discriminate on the basis of race, religion, color, national origin, sex, sexual orientation, age, veteran status, disability, genetic information, or other applicable legally protected characteristic. Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable law, including the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Unincorporated LA County workers: we reasonably believe that criminal history may have a direct, adverse and negative relationship with the following job duties, potentially resulting in the withdrawal of a conditional offer of employment: client provided property, including hardware (both of which may include data) entrusted to you from theft, loss or damage; return all portable client computer hardware in your possession (including the data contained therein) upon completion of the assignment, and; maintain the confidentiality of client proprietary, confidential, or non-public information. In addition, job duties require access to secure and protected client information technology systems and related data security obligations.

Salary : $39 - $61