Senior Machine Learning Engineer (AI)

TL;DR

Senior Machine Learning Engineer (AI): Building and optimizing evaluation frameworks and synthetic data pipelines for healthcare-specific LLMs with an accent on safety, clinical accuracy, and automated benchmarking. Focus on developing LLM-as-judge systems, analyzing failure modes, and scaling human-in-the-loop evaluation.

Location: Must be based in Palo Alto and work from the office five days a week

Company

Leading generative AI company in healthcare developing safety-focused LLMs for autonomous clinical conversations with patients.

What you will do

• Design and build evaluation frameworks for LLM safety, clinical accuracy, and conversational quality.
• Develop synthetic data generation pipelines to stress-test models across diverse clinical scenarios.
• Build automated and human-in-the-loop evaluation pipelines at scale.
• Create benchmarks, metrics, and LLM-as-judge systems for healthcare tasks.
• Analyze failure modes and collaborate with the post-training team to drive model improvements.
• Partner with research, engineering, and clinical teams to define and raise the quality bar.

Requirements

• MS or PhD in Computer Science or a related field.
• 5+ years of experience in ML engineering, evaluation systems, or applied ML.
• Strong software engineering skills in Python and PyTorch with production-quality code.
• Hands-on experience with LLM evaluation, benchmarking, or synthetic data generation.
• Ability to build robust data analysis and evaluation infrastructure.
• Experience with UI/UX and front-end tools such as Streamlit, Gradio, or React.

Nice to have

• Experience in healthcare AI.
• Experience with RL/RLVR/RLHF or safety evaluation.

Culture & Benefits

• Opportunity to work at a high-growth startup backed by top-tier AI and healthcare investors.
• Collaboration with a world-class team from institutions like Stanford, Google, Meta, and NVIDIA.
• Participation in category creation within the healthcare AI domain.
• Strong in-person team culture based in Palo Alto.