Senior Manager, AI Performance & Operations (AI)

TL;DR

Senior Manager, AI Performance & Operations (AI/MLOps): Ensuring the stability and reliability of production AI agentic workflows with an accent on system monitoring, incident response, and evaluation frameworks. Focus on diagnosing failures across models, prompts, and RAG pipelines, and building scalable LLM-as-judge methodologies.

Location: Remote (Must be based in the United States)

Salary: $110,000–$172,000 per year

Company

A healthcare-focused product company developing high-impact AI agentic workflows to improve patient and user outcomes.

What you will do

• Own the post-launch performance and operational health of high-volume AI workflows, ensuring quality and safety.
• Design and oversee production monitoring dashboards to track containment, escalation rates, and user impact.
• Lead diagnosis and resolution of AI incidents and regressions across models, prompts, and orchestration layers.
• Build scalable evaluation frameworks using golden datasets and LLM-as-judge methodologies.
• Drive continuous improvement cycles through prompt tuning, experimentation, and system iteration.
• Collaborate with cross-functional stakeholders to communicate system performance, risks, and resolutions.

Requirements

• 7+ years of experience in applied AI, ML operations, or product analytics.
• Hands-on experience with LLMs, agentic systems, and RAG pipelines in production environments.
• Proven ability to design scoring rubrics, regression testing, and sampling frameworks.
• Deep diagnostic skills to isolate failures in retrieval systems and workflow logic.
• Must be based in the United States.
• Experience in regulated or high-reliability domains such as healthcare is a plus.

Culture & Benefits

• Competitive salary range with stock options for all employees.
• Comprehensive employer-sponsored medical, dental, and vision coverage.
• 401(k) retirement savings program.
• Generous paid time off and parental leave programs.
• Monthly $100 stipend for mobile or internet expenses.
• Flexible, mission-driven work environment focused on improving healthcare.