Senior AI Platform Engineer – QE Automation

GenAI GenOS Engineer (Senior)

About the Team

GenOS is a Generative AI Operating System — the platform every GenAI experience is built, deployed, and governed on, including customer-facing AI assistants.

The stack includes:

• GenStudio — LLM sandbox and extensible model catalog; new models onboarded in days.
• AI Workbench — versioned Prompt Management and the LLM Leaderboard for benchmarking.
• GenRuntime — GenOrchestrator (planner, executor, memory, retrieval) plus agents and tools grounding LLMs in domain knowledge.
• GenUX — 140 AI UX components consumed by product teams.
• GenSRF — security, risk, and fraud guardrails as a platform feature, not an afterthought.

Multi-LLM catalog:

• Anthropic Claude via AWS Bedrock
• Gemini
• Llama
• Mistral

The Problem This Role Exists to Solve

Apps built through GenOS ship with an AI-accelerated development innerloop — and no generated quality assets.

Development is agentic end-to-end; QE starts from zero, manually, after the fact. Testing is now the bottleneck consuming the platform's velocity gains.

This role makes QE a native GenOS output: build an app on the platform, and its test agents, suites, and release gates come with it — self-serve, on the paved road, like every other capability.

Role Summary

Senior, expert-level, embedded hands-on engineer building QE enablement as a GenRuntime platform capability.

You build the agents and platform hooks that automatically emit test assets and quality gates for every app scaffolded through GenOS.

You ship production code weekly under Engineering Lead direction.

This is a build role — the deliverable is platform capability, not test execution, not advisory, not architecture-on-slides.

Key Responsibilities

• Build QE-generation agents on GenRuntime: agents that consume artifacts the GenOS dev flow already produces — requirements, code diffs, API specs, GenUX component usage — and emit functional, API, and regression test assets as validated, structured output.
• Wire QE enablement into the paved road: an app scaffolded through GenOS gets QE agents attached by default — no opt-in ceremony, self-serve onboarding, zero hand-holding.
• Build the agent toolbelt as reusable GenRuntime tools: test execution, self-healing selectors and API contracts, failure triage, defect summarization — including agent-to-agent patterns where test agents interrogate the application's own agents.
• Extend AI Workbench eval primitives (LLM Leaderboard, prompt evaluation) into release gates for GenAI applications: golden datasets, LLM-as-judge scoring, statistical quality thresholds enforced in paved-road CI/CD — not just model selection.
• Embed GenSRF coverage into generated tests: safety, privacy, and moderation regressions become executable test cases, not audit findings.
• Integrate with Feature Management so QE-agent rollout is flagged, measured, and adoption-tracked per product team.
• Write well-tested, production-grade code; participate in reviews and design discussions — PR merge velocity and AI-assisted code in PRs are tracked org KPIs.
• Participate in the production support/on-call rotation for the QE capability surface (rotation shape and compensation treatment per Open Items).
• Contribute self-serve onboarding docs and inner-source repos — inner-source contribution is a tracked metric.

Must-Have Qualifications

• 7 years backend/platform engineering (Java, Python, or Go) with deep distributed-systems fundamentals: async processing, caching, idempotency, failure handling.
• 2–3 years building LLM-powered systems in production, not prototypes: agent/orchestration frameworks (LangChain / LlamaIndex / homegrown), structured tool calling, prompt versioning and evaluation.
• Structured-output engineering at production grade — generated tests are code artifacts: schema enforcement, output validation, and repair loops are the daily job.
• Demonstrated QE domain depth: has built or owned test automation architecture — framework design, CI quality gates, flaky-test economics — enough to encode that judgment into agent behavior. A platform engineer who has never owned a test suite will build QE agents that generate garbage confidently.
• One major LLM provider at scale — AWS Bedrock strongly preferred — with real operational scars: rate limits, latency variance, provider failover, version drift.
• AWS Kubernetes deployment depth (services run on Kubernetes Service).
• Fluent in AI-assisted development workflows — Copilot-class tooling is the expected daily working mode.

Nice-to-Have

• LLM evaluation engineering: golden datasets, LLM-as-judge calibration, mutation testing or fault injection to validate generated-test quality.
• MCP tool integrations; SSE/WebSocket streaming for agentic responses.
• Vector stores and retrieval in production (OpenSearch / pgvector / Pinecone or equivalent).
• Multi-tenant enterprise platforms under strict security/compliance; fintech background.
• Open-source / inner-source contribution record.