Senior Manager, AI Performance & Operations (AI)

TL;DR

Senior Manager, AI Performance & Operations (AI): Own production performance and continuous improvement of Devoted’s agentic AI workflows for service and experience, with an accent on monitoring, incident response, and measurable optimization. Focus on designing scalable evaluation systems, running structured mini-launch experiments (model/prompt/tool changes), and ensuring safe handoff into general availability.

Company

hirify.global builds an all-in-one healthcare solution combining insurance, clinical care, service, and technology.

What you will do

• Own post-launch accountability for high-volume, high-risk voice and text AI workflows, ensuring business, quality, safety, and operational outcomes.
• Maintain a portfolio view of workflow status, maturity, failure modes, and next actions; detect degradation early.
• Monitor weekly indicators (containment, escalation, complaints) and oversee production dashboards to track behavior.
• Lead AI regression diagnosis and incident response, distinguishing model, prompt, or orchestration issues and driving post-mortems to closure.
• Run continuous improvement cycles (hypothesis, testing, rollout/rollback) for mini-launches such as model swaps, prompt changes, and tool updates.
• Design evaluation systems (golden datasets, LLM-as-judge, human review queues) and guide early-stage workflows through safe instrumentation and launch criteria.

Requirements

• Strong production experience with LLMs, RAG, voice AI, agentic orchestration, or equivalent ML systems.
• Experience designing or operating evaluation systems: rubrics, scoring frameworks, regression tests, and production-sampling workflows.
• Ability to diagnose failure modes across model reasoning, tool failure, retrieval issues, and workflow logic.
• Comfort with active experimentation, including prompt/context tuning, retrieval redesign, and safe model migrations.
• Excellent cross-functional communication and judgment in ambiguous, high-accountability environments.
• Healthcare or regulated workflow experience (e.g., contact center operations or high-stakes service quality/reliability).

Culture & Benefits

• Employer-sponsored health, dental, and vision plans with low or no premium.
• Generous paid time off and parental leave program.
• $100 monthly mobile or internet stipend.
• Stock options for all employees.
• 401K program and bonus eligibility (excluding Director and above).

Hiring process

• Interview process with compensation details provided by a talent partner after it begins.

Location: Remote (USA)

Salary: $110,000 - $172,000 / year