Research Scientist - Benchmarks & Evals (AI)

Location: United States; remote

Base salary: $150,000–$250,000 per year; equity: $350,000–$750,000

Company

A consumer application and human-data network collecting opt-in photos, videos, and documents for rights-aware AI training data.

What you will do

• Design and ship public benchmarks for image, video, and document models.
• Build private test sets from real opt-in data that models have not trained on.
• Run data-value experiments comparing models trained with and without proprietary data.
• Design human evaluation studies and automatic metrics.
• Publish leaderboards and research reports for frontier AI labs and data buyers.
• Identify model weaknesses and translate findings into recommendations for future data collection and licensing.

Requirements

• 3+ years of machine learning research or research engineering experience.
• A benchmark or evaluation used by other practitioners.
• Hands-on experience training and fine-tuning image or video models with PyTorch.
• Strong understanding of experimental design and statistics, including controls, ablations, and human studies.
• Experience running experiments end to end, from cloud GPUs to final report.
• Clear technical writing for researchers and data buyers.

Nice to have

• Publications at NeurIPS, ICML, ICLR, or CVPR, especially on datasets and benchmarks.
• Experience on an evaluation or data team at an AI lab.
• Experience selling or licensing data to AI labs.
• Experience with data valuation or data attribution research.

Culture & Benefits

• Work on applied AI data infrastructure operating at large scale.
• Own systems supporting hundreds of millions of opt-in photos, videos, and documents.
• Build infrastructure influencing the data used to train next-generation AI models.
• Work with millions of uploads per day and contributors worldwide.
• Current stack includes Python, PyTorch, open-weight image and video models, cloud GPUs, PostgreSQL via Supabase, and S3 storage.