Member of Engineering (Evaluations) (AI)

TL;DR

Member of Engineering (Evaluations) (AI): Design and implement infrastructure and tooling for researchers and engineers with an accent on evaluations and benchmarks for base models and instruction-following models. Focus on researching and implementing meaningful metrics for real-world software development skills and collaborating with applied research and product teams.

Location: Remote (EMEA/East Coast)

Company

Building AGI to power economically valuable work and scientific progress, with a remote-first team across Europe and North America.

What you will do

• Research and implement evaluations and benchmarks for base models and instruction-following models
• Collaborate with applied research and product teams to define meaningful metrics capturing progress on real-world software development skills
• Work in a team: plan future steps, discuss ideas, and communicate clearly with peers

Requirements

• Strong experience with Large Language Models (LLMs), including understanding and intuition of their limitations
• Strong engineering background: programming skills across multiple languages including Python, familiarity with full software development lifecycle
• Experience with Linux, strong algorithmic skills, use of modern tools
• Good taste, curious mindset, strong critical thinking, and ability to question code quality policies

Culture & Benefits

• Fully remote work and flexible hours
• 37 days/year of vacation and holidays
• Health insurance allowance for you and dependents
• Company-provided equipment, wellbeing, always-be-learning, and home office allowances
• Frequent team get-togethers: monthly 3-day in-person meetings and longer offsites twice a year
• Great diverse and inclusive people-first culture

Hiring process

• Intro call with one of our Founding Engineers
• Technical interview(s) with one of our Founding Engineers
• Team fit call with the People team
• Final interview with one of our Founding Engineers