Machine Learning Research Scientist, Evaluations (AI)

Company

Scale AI develops high-quality data and full-stack technologies that support leading AI models and enterprise and government AI applications.

What you will do

• Analyze frontier LLM and agent behavior to identify, characterize, and diagnose capability gaps, reasoning errors, robustness issues, and alignment failures.
• Design and build benchmarks and evaluation methods for text and multimodal models.
• Apply SFT, RLHF, preference modeling, reward modeling, and instruction-tuning expertise to connect model failures with data and training interventions.
• Collaborate with researchers, engineers, and foundation model labs to establish evaluation-driven AI development practices.
• Translate failure analysis into technical and strategic input for future generative AI models.
• Publish research findings at major AI conferences and journals.

Requirements

• Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
• Experience with LLM evaluation, benchmark development, and post-training techniques such as RLHF, preference modeling, or instruction tuning.
• Excellent written and verbal communication skills.
• Published machine learning research in major conferences or journals.
• Previous experience in a customer-facing role.

Culture & Benefits

• Base salary, equity, and benefits may be included for eligible roles.
• Comprehensive health, dental, and vision coverage.
• Retirement benefits, a learning and development stipend, and generous paid time off.
• Additional benefits may include a commuter stipend.
• Inclusive equal opportunity workplace with reasonable accommodations available during the application and recruiting process.

Hiring process

• Compensation is determined during the interview process based on work location, skills, experience, qualifications, interview performance, and education or training.
• Candidates must wait 90 days before being reconsidered for the same role.