Manager, AI Quality Evals (AI)

TL;DR

Manager, AI Quality Evals (AI): Leading the human evaluation system for AI product quality with an accent on labeling operations, rubric design, and quality reporting. Focus on building a high-leverage team to ensure realistic datasets and measurable quality lift for agentic AI capabilities.

Location: Hybrid (4 days a week in Mountain View, CA)

Salary: $195,000 - $240,000

Company

hirify.global is an AI platform providing enterprise search, AI assistants, and scalable AI agents to help organizations manage and act on knowledge.

What you will do

• Lead end-to-end operational evaluation across human labeling and benchmarking.
• Manage a senior team focused on answer quality and recurring labeling operations.
• Establish rhythms for weekly triage, eval health reviews, and quality reporting.
• Evaluate performance across priority workflows, including tool use and response quality.
• Partner with the eval team to curate datasets and improve LLM-as-a-judge raters.
• Own labeling guidelines and calibration to debias LLM judges against human labels.

Requirements

• 7+ years of experience in evaluation operations, ML data ops, or technical program management.
• Experience leading or mentoring high-performing individual contributors.
• Deep familiarity with AI product evaluation, rubric design, and human labeling systems.
• Experience with LLM-as-a-judge systems and model-eval tooling.
• Strong analytical judgment to identify failure patterns and turn them into actionable items.
• Must be based in or able to work hybrid (4 days/week) in Mountain View, CA

Culture & Benefits

• Comprehensive medical, vision, and dental coverage.
• 401k plan and generous time-off policy.
• Home office improvement stipend and annual education and wellness stipends.
• Vibrant company culture with regular events and daily healthy lunches.

Hiring process

• Includes a brief AI-focused exercise or discussion to evaluate design and usage of AI to drive impact.