What are the responsibilities and job description for the Technical Program Manager, Human Evaluation Operations position at Microsoft AI?
Overview
Microsoft AI (MAI) is building the world’s most advanced AI systems—and rigorous, scalable human evaluation is foundational to ensuring our models are safe, aligned, and high‑quality. The Human Evaluation Operations (Human Eval Ops) team powers this by running one of the largest and most reliable human‑in‑the‑loop pipelines at Microsoft.
We are hiring two Technical Program Managers to join this team and own end‑to‑end evaluation operations for model quality, safety, and capability development. These TPMs will partner closely with product squads, engineering, data scientists, researchers, and external annotation vendors to deliver high‑quality human evaluations at scale.
You will drive programs that ensure MAI has the people, processes, training pipelines, and tooling needed to enable fast, trustworthy, and efficient evaluation across a wide range of AI tasks.
This is a highly cross‑functional, execution‑oriented TPM role ideal for someone who thrives in operational complexity, is deeply organized, and loves working at the intersection of people, process, and product quality.
Responsibilities
Required Qualifications
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
Microsoft AI (MAI) is building the world’s most advanced AI systems—and rigorous, scalable human evaluation is foundational to ensuring our models are safe, aligned, and high‑quality. The Human Evaluation Operations (Human Eval Ops) team powers this by running one of the largest and most reliable human‑in‑the‑loop pipelines at Microsoft.
We are hiring two Technical Program Managers to join this team and own end‑to‑end evaluation operations for model quality, safety, and capability development. These TPMs will partner closely with product squads, engineering, data scientists, researchers, and external annotation vendors to deliver high‑quality human evaluations at scale.
You will drive programs that ensure MAI has the people, processes, training pipelines, and tooling needed to enable fast, trustworthy, and efficient evaluation across a wide range of AI tasks.
This is a highly cross‑functional, execution‑oriented TPM role ideal for someone who thrives in operational complexity, is deeply organized, and loves working at the intersection of people, process, and product quality.
Responsibilities
- Lead Human Evaluation Programs: Drive end‑to‑end human evaluation workflows supporting model quality, safety, and capability initiatives across MAI. Coordinate evaluation planning, task design alignment, and delivery with product squads, engineering, and research partners.
- Manage Evaluation Workforce & Readiness: Oversee the health, performance, and scalability of MAI’s human evaluation workforce—including onboarding, qualification, training, and continuous performance management—to ensure reliable, high‑quality evaluation signals.
- Operational Excellence & Quality Governance: Maintain high operational standards across human‑in‑the‑loop pipelines by monitoring quality signals, resolving issues, and guiding teams toward consistent, trustworthy evaluation outcomes.
- Cross‑Functional Program Leadership: Partner with product squads to scope evaluation needs, define instructions and scorecards, support experimentation, and ensure teams are equipped to use human evaluations effectively.
- Platform & Vendor Partnership Management: Represent MAI needs to platform and vendor partners, shaping their roadmaps and ensuring capacity, reliability, and compliance with MAI standards.
- Insights, Tooling, & Documentation: Provide evaluation insights to product teams, maintain essential documentation, and influence the evolution of internal tools, dashboards, and processes that enable scalable human evaluations.
- Specialized Evaluation Programs: Support domain‑specific or advanced evaluation initiatives (e.g., expert reviews, structured scoring programs) in collaboration with MAI stakeholders.
Required Qualifications
- Bachelor's Degree AND 2 years experience in engineering, product/technical program management, data analysis, or product development OR equivalent experience.
- 1 year(s) of experience managing cross-functional and/or cross-team projects.
- 3 years of technical program management, operations management, data operations, or equivalent experience
- 1 year(s) of experience reading and/or writing code (e.g., sample documentation, product demos).
- Experience working cross‑functionally with engineering, research, PM, vendors, and operations partners.
- Experience managing vendor relations or external workforce programs.
- Strong analytical skills and comfort working with dashboards, metrics, and evaluation data.
- Experience running human‑in‑the‑loop data pipelines (e.g., annotations, RLHF, safety evals, quality assurance, crowdsourcing).
- Familiarity with LLM and AI model evaluation practices, data annotation platforms and systems.
- Ability to quickly understand product quality signals, debug task design issues, and iterate with engineering teams.
- Experience in operational excellence, process automation, or scaling manual workflows.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
Salary : $100,600 - $215,400