AI Evaluation Systems Engineering Lead

Location: Mountain View, United States

Salary: USD $142,800–$274,800 per year across the U.S. for Software Engineering IC5; location-specific ranges apply in the San Francisco Bay Area and New York City metropolitan area. Software Engineering IC6 ranges are USD $165,600–$296,400 per year nationally and USD $220,800–$331,200 in those locations.

Company

Microsoft AI develops artificial intelligence products, research, and infrastructure.

What you will do

• Set the architecture, technical direction, roadmap, and delivery strategy for the Eval Systems team.
• Build end-to-end evaluation infrastructure for benchmark configuration, model inference, grading, aggregation, and results publication.
• Improve evaluation turnaround time, scheduling, throughput, and GPU efficiency for research iteration.
• Establish reproducibility, traceability, validation, regression checks, observability, and failure recovery.
• Contribute code, lead design reviews, and debug distributed evaluation, inference, and execution problems.
• Hire, mentor, and develop engineers while coordinating priorities and tradeoffs with research and engineering partners.

Requirements

• Bachelor’s degree in Computer Science or a related technical discipline and 6+ years of technical engineering experience with coding, or equivalent experience.
• Experience with production distributed systems, infrastructure platforms, or ML systems.
• Strong programming skills, including Python and experience with large shared codebases.
• Experience diagnosing distributed-system failures and improving reliability, performance, resource efficiency, monitoring, incident response, and on-call practices.
• Technical leadership experience, including setting direction, delivering complex projects, mentoring engineers, and collaborating with research and engineering teams.
• Experience with LLM evaluation, benchmarking, model-based grading, reinforcement learning infrastructure, GPU-backed inference, distributed training, or large-scale compute workloads.

Nice to have

• 10+ years of technical engineering experience.
• Familiarity with Ray, Kubernetes, or Slurm.
• Experience improving experimental reproducibility, validating metrics, or investigating differences across evaluation and serving environments.
• Experience hiring engineers, managing people, or growing a technical team.

Culture & Benefits

• Focus on clear ownership, collaboration, technical excellence, and inclusive team development.
• Benefits and other compensation may be available depending on the role and location.
• Applications are accepted on an ongoing basis until the position is filled.