Research Engineer, Model Evaluations (AI)

Location: Remote-friendly, with required travel and at least 25% office attendance in San Francisco, CA or New York City, NY

Annual salary: $500,000–$850,000 USD

Company

Anthropic develops reliable, interpretable, and steerable AI systems designed to be safe and beneficial.

What you will do

• Design and run evaluations covering Claude's reasoning, agentic behavior, knowledge, and safety properties.
• Build and operate distributed evaluation infrastructure that runs reliably against live training checkpoints at scale.
• Develop dashboards and visualizations for monitoring model health, regressions, latency, and signal quality.
• Debug anomalous evaluation results during training and distinguish model changes from data, harness, or infrastructure issues.
• Improve evaluation tooling, libraries, workflows, observability, and experiment-tracking systems.
• Partner with research teams to define measurable capabilities, run experiments, and communicate results to technical and non-technical audiences.

Requirements

• Strong Python programming skills, including production or research infrastructure.
• Experience building or operating reliable distributed systems, data pipelines, or large-scale infrastructure.
• Clear written and verbal communication skills, including explaining technical results to non-specialists.
• Comfort working in an on-call or production-support capacity during live training runs.
• Bachelor's degree or equivalent education, training, or experience in a relevant field.
• Interest in the societal impacts of AI and in developing safe, beneficial systems.

Nice to have

• Experience with large language models, prompting, sampling, and scaffolding.
• Experience with data visualization, trusted dashboards, language-model evaluation metrics, or observability systems.
• Background in statistics, experimental design, large-scale dataset processing, or ML training infrastructure.
• Experience collaborating across research teams and pair programming.

Culture & Benefits

• Collaborative research environment focused on large-scale empirical AI research.
• Frequent research discussions and cross-functional collaboration.
• Flexible working hours, generous vacation and parental leave, and competitive benefits.
• Optional equity donation matching and office space for collaboration.
• Visa sponsorship is available, with immigration-lawyer support, subject to role and candidate eligibility.