Member of Engineering (Evaluations / Engineering) (AI)

TL;DR

Member of Engineering (Evaluations / Engineering) (AI): Build a scalable self-serve evaluation platform to power research and development with an accent on distributed evaluations, benchmarking frameworks, and experimentation tooling. Focus on designing Python frameworks for benchmarks, implementing large-scale pipelines, and collaborating with modeling and product teams.

Location: Remote (EMEA/East Coast)

Company

Building Artificial General Intelligence to power economically valuable work and scientific progress by stacking advantages in applied research, engineering, infrastructure, and model scaling.

What you will do

• Design a Python framework for easy implementation of internal and public benchmarks in a centralized way.
• Build and maintain pipelines for running distributed evaluations at scale.
• Collaborate with modeling and product teams to identify and improve experimentation and evaluation tooling.

Requirements

• Strong engineering background: experience leading cross-functional software projects and building highly reliable, well-tested services.
• Experience with distributed systems: data pipelines, distributed processing, message queues and event-driven architectures (e.g., Kafka, Google Pub/Sub), clouds (GCP, AWS, Azure), monitoring and alerting (Grafana, Prometheus, Datadog).

Nice to have

• Experience designing frameworks or tooling for developers.
• Product mindset towards building developer-facing software.
• Knack for collaborating with researchers and ML engineers.
• Experience with MLOps and data visualization platforms.

Culture & Benefits

• Fully remote work with flexible hours.
• Remote-first team across Europe and North America with monthly 3-day in-person meetups and biannual offsites.
• 37 days/year of vacation & holidays.
• Health insurance allowance for you and dependents, company-provided equipment, wellbeing, learning, and home office allowances.
• Diverse & inclusive people-first culture.

Hiring process

• Intro call with a Founding Engineer.
• Technical interview(s) with a Founding Engineer.
• Team fit call with the People team.
• Final interview with a Founding Engineer.