Software Engineer | Health Evaluations (AI)

Location: Hybrid in New York City, United States, with regular on-site working sessions in Rochester, Minnesota, and travel to London.

Salary: USD $188,000–$304,200 per year in the New York City metropolitan area.

Company

Microsoft AI develops AI products and is applying them to healthcare through a partnership with Mayo Clinic.

What you will do

• Own the end-to-end evaluation strategy for Mayo Clinic AI agents, defining metrics, datasets, benchmarks, and quality-improvement actions.
• Build static, multi-turn, adversarial, grounded, and agentic evaluations covering tools, retrieval, connected data, trajectories, and final responses.
• Create clinically grounded guidelines, case-specific rubrics, LLM-as-judge prompts, automated evaluations, and clinician-calibrated rating systems.
• Evaluate patient-facing experiences for factuality, groundedness, safety, escalation, health literacy, and safeguards for psychologically vulnerable users.
• Develop scalable evaluation pipelines and dashboards, monitor production interactions with privacy-preserving processes, and support launch or no-launch decisions.
• Coordinate engineering, research, data science, and Mayo clinical and IT stakeholders while handling sensitive health data under HIPAA and partner governance requirements.

Requirements

• Bachelor’s degree in Computer Science or a related technical field, plus 6+ years of technical engineering experience or equivalent experience.
• Coding experience in languages including C, C++, C#, Java, JavaScript, or Python.
• 3+ years leading multidisciplinary projects, including requirements definition, planning, and cross-functional execution.
• Experience with software development, data science, machine learning, or LLM-powered products.
• Ability to work in a hybrid arrangement based in New York City and attend regular on-site sessions in Rochester, Minnesota.
• Ability to handle sensitive health data responsibly within HIPAA and partner data-governance requirements.

Nice to have

• Experience in healthcare, health technology, regulated, or safety-critical domains.
• Experience with LLM training and evaluation, LLM-as-judge design, user simulation, and agentic or trajectory evaluation.
• Familiarity with clinical workflows, medical terminology, clinician stakeholders, HIPAA, PHI handling, FDA guidance, or the EU AI Act.
• Experience operating within partner organizations and translating between clinical, technical, research, and business contexts.

Culture & Benefits

• Embedded collaboration with Mayo Clinic clinicians, subject-matter experts, and technical stakeholders.
• Cross-team work with MAI engineering, research, data science, and health evaluation teams.
• Regular travel to Mayo Clinic in Rochester and the MAI Health Evaluations team in London.
• Benefits and additional compensation may be available depending on role eligibility.