Member Of Technical Staff - Evaluations (AI)

TL;DR

Member of Technical Staff - Evaluations (AI): Conducting critical comparative analysis and building robust evaluation systems for open-weight AI models with an accent on reasoning, alignment, and model usefulness. Focus on developing generalizable frameworks, designing experimental pipelines, and collaborating with training teams to accelerate model improvement feedback loops.

Location: On-site in San Francisco, London, or New York

Company

An AI research startup building open-weight superintelligence systems by leveraging talent from industry-leading organizations like OpenAI, DeepMind, and Meta.

What you will do

• Conduct comparative analysis to understand and enhance model capabilities.
• Build and refine evaluation processes to bridge data, evals, and model behavior.
• Develop generalizable evaluation frameworks for reasoning and alignment.
• Collaborate with pre-training and applied teams to translate insights into improvements.
• Measure model progress using synthetic evals, human feedback, and real-world interaction data.

Requirements

• Strong background in statistical analysis and experimental design.
• Experience with LLM evaluation methodologies including static benchmarks and human preference evals.
• Ability to thrive in a fast-paced startup environment.
• Detail-oriented approach to building complex feedback loops.
• High agency and collaborative mindset for cross-functional research.

Culture & Benefits

• Top-tier compensation package including salary and equity.
• Comprehensive medical, dental, vision, life, and disability insurance.
• Fully paid parental leave for all new parents including family planning support.
• Daily provided lunch and dinner in the office.
• Regular team off-sites and celebrations.