Applied AI Engineer, Evaluations, Health (AI)

Location: Hybrid in New York, United States, with regular travel to Mayo Clinic in Rochester, Minnesota, and to London

Salary: USD $188,000–$304,200 per year in the New York City metropolitan area; the general U.S. range is USD $142,800–$274,800 per year

Company

Microsoft AI develops AI products, including healthcare agents created with Mayo Clinic to support patients and clinicians.

What you will do

• Own the end-to-end evaluation strategy for Mayo Clinic healthcare agents, defining metrics, datasets, quality targets, and improvement actions.
• Build layered evaluations covering single-turn benchmarks, multi-turn user simulations, adversarial scenarios, tool use, retrieval, connected data, trajectories, and final responses.
• Work with clinicians and subject-matter experts to create clinical guidelines, case-specific rubrics, clinician-authored test sets, and calibrated physician review programs.
• Design LLM-as-judge prompts and automated evaluation pipelines, measuring agreement between automated judges, clinicians, and human raters.
• Evaluate factuality, groundedness, safety, escalation, health literacy, and safeguards for psychologically vulnerable users in patient-facing AI experiences.
• Partner with engineering, research, data science, and Mayo stakeholders on dashboards, production monitoring, quality triage, lifecycle oversight, and launch decisions.

Requirements

• Bachelor’s degree in Computer Science or a related technical field and 6+ years of technical engineering experience, or equivalent experience.
• Coding experience with languages including C, C++, C#, Java, JavaScript, or Python.
• 3+ years leading multidisciplinary projects, including requirements definition, planning, and cross-functional execution.
• Experience with software development, data science, machine learning, or evaluation of ML- or LLM-powered products.
• Ability to work with clinicians and technical stakeholders in a regulated, safety-critical healthcare environment.
• Ability to handle sensitive health data under HIPAA and partner data-governance requirements.

Nice to have

• Healthcare, health tech, regulated-domain, clinical workflow, medical terminology, or clinician-stakeholder experience.
• Experience with LLM-as-judge design, user simulation, agentic evaluation, or trajectory evaluation.
• Experience with HIPAA, PHI handling, FDA guidance for AI-enabled devices, predetermined change-control plans, or the EU AI Act.
• Experience operating inside a partner organization and translating partner needs into technical requirements.

Culture & Benefits

• Hybrid work with regular in-person collaboration in New York, Rochester, and London.
• Work at the intersection of product, research, clinical practice, engineering, and data science.
• Focus on empathy, kindness, initiative, positive energy, and comfort with ambiguity.
• Opportunity to influence the safety and quality of healthcare AI used by patients and clinicians.