Machine Learning Research Scientist, Evaluations (AI)

Company

Scale AI develops high-quality data and full-stack technologies that support leading AI models and enterprise and government AI applications.

What you will do

• Analyze frontier LLM and agent behavior to identify, characterize, and diagnose capability, reasoning, robustness, and alignment failures.
• Design and build benchmarks and evaluation methods for text and multimodal models.
• Apply supervised fine-tuning, reinforcement learning from human feedback, and reward modeling to connect observed failures with data and training interventions.
• Collaborate with researchers and engineers to establish evaluation-driven AI development practices.
• Translate failure analysis into technical and strategic input for foundation model partners.
• Publish research findings in leading AI conferences and journals.

Requirements

• Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
• Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
• Experience with RLHF, preference modeling, instruction tuning, LLM evaluation, or benchmark development.
• Excellent written and verbal communication skills.
• Research publications in machine learning at major conferences or journals.
• Previous experience in a customer-facing role.

Culture & Benefits

• Health, dental, and vision coverage.
• Retirement benefits, learning and development stipend, and generous paid time off.
• Potential commuter stipend depending on role eligibility.
• Eligible roles may include equity compensation subject to approval.
• Inclusive equal-opportunity workplace with reasonable accommodation support.

Hiring process

• Salary placement is determined during the interview process based on work location, skills, experience, qualifications, interview performance, and education or training.
• Candidates must wait 90 days before being reconsidered for the same role.

Salary: $165,600–$207,000 USD per year, with potential equity and benefits.