Technical Program Manager, Human Evaluation Operations (AI)

TL;DR

Technical Program Manager (AI): Driving end-to-end human evaluation operations for Microsoft's advanced AI systems with an accent on model quality, safety, and scalability. Focus on managing cross-functional evaluation programs, workforce readiness, and operational excellence within human-in-the-loop pipelines.

Location: Onsite in Redmond, United States

Salary: $100,600 – $199,000 USD per year (typical U.S. base pay), with a range of $131,400 – $215,400 USD per year for San Francisco Bay area and New York City metropolitan area.

Company

hirify.global is building the world’s most advanced AI systems, focusing on rigorous and scalable human evaluation to ensure models are safe, aligned, and high-quality.

What you will do

• Lead end-to-end human evaluation programs for AI model quality, safety, and capability initiatives.
• Oversee the health, performance, and scalability of the human evaluation workforce, including onboarding, qualification, and training.
• Maintain high operational standards and quality governance across human-in-the-loop pipelines.
• Partner with product squads, engineering, and research to scope evaluation needs and ensure effective use of human evaluations.
• Represent MAI needs to platform and vendor partners, shaping their roadmaps and ensuring compliance.
• Provide evaluation insights to product teams and influence the evolution of internal tools and processes for scalable human evaluations.
• Support specialized evaluation initiatives, such as expert reviews and structured scoring programs.

Requirements

• Bachelor’s Degree AND 2+ years experience in engineering, product/technical program management, data analysis, or product development OR equivalent experience.
• 1+ year(s) of experience managing cross-functional and/or cross-team projects.

Nice to have

• 3+ years of technical program management, operations management, or data operations experience.
• 1+ year of experience reading and/or writing code.
• Experience working cross-functionally with engineering, research, PM, vendors, and operations partners.
• Experience managing vendor relations or external workforce programs.
• Strong analytical skills and comfort working with dashboards, metrics, and evaluation data.
• Experience running human-in-the-loop data pipelines (e.g., annotations, RLHF, safety evals).
• Familiarity with LLM and AI model evaluation practices, data annotation platforms and systems.
• Ability to quickly understand product quality signals, debug task design issues, and iterate with engineering teams.
• Experience in operational excellence, process automation, or scaling manual workflows.

Culture & Benefits

• Microsoft is an equal opportunity employer committed to diversity and inclusion.
• All qualified applicants receive consideration for employment without regard to age, race, gender, disability, or other protected characteristics.
• Assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process is available.
• Certain roles may be eligible for benefits and other compensation (details available on the Microsoft Careers website).

Hiring process

• This position will be open for a minimum of 5 days.
• Applications are accepted on an ongoing basis until the position is filled.