Senior Machine Learning Engineer (AI)

TL;DR

Senior Machine Learning Engineer (AI): Building and scaling observability, evaluation, and improvement loops for production-grade agentic AI systems with an accent on reliability, drift detection, and performance validation. Focus on designing robust ML infrastructure, orchestrating multi-agent systems, and bridging the gap between experimental research and production deployment.

Location: Must be based in the United States

Company

Scale AI is a leading AI data foundry providing high-quality data and full-stack technologies to power advanced models for enterprises and governments.

What you will do

• Build observability tools to monitor agent behavior and performance in production environments.
• Design and automate evaluation methodologies and metrics for agentic applications at scale.
• Develop and ship ML systems to detect drift, anomalies, and misalignment in agent behavior.
• Design and execute rigorous experiments to validate model and agent performance improvements.
• Collaborate with cross-functional teams to translate enterprise requirements into robust platform capabilities.
• Take ownership of ML systems from initial prototype through to reliable production deployment.

Requirements

• 5+ years of experience as an ML engineer or applied scientist on production LLM-powered systems.
• Strong grounding in building evaluation, monitoring, or continuous-learning infrastructure for ML systems.
• Hands-on experience with LLMs and agent architectures including tool use, planning, and multi-agent orchestration.
• Proven ability to partner with software engineers to productionize research and experimental work.
• Rigorous approach to experimentation with clear hypotheses and statistical grounding.
• Track record of collaborating across functions to navigate ambiguous requirements.

Nice to have

• Experience with RLHF, SFT, reward modeling, or verifiable-reward systems.
• Experience with model or systems optimization regarding latency, cost, or inference efficiency.
• Published research, open-source contributions, or patents in agentic systems or applied ML.
• Experience working in regulated or enterprise contexts.

Culture & Benefits

• Comprehensive health, dental, and vision coverage.
• Retirement benefits and equity-based compensation.
• Learning and development stipend.
• Generous PTO policy.
• Inclusive and equal opportunity workplace.