Data Ingestion Lead (AI)

TL;DR

Data Ingestion Lead (AI): Leading the team responsible for building and scaling the ingestion layer for frontier model training corpora with an accent on web crawling, extraction pipelines, and data lake architecture. Focus on mentoring a high-performing engineering team, driving technical strategy for multi-TB scale data acquisition, and partnering with research teams to optimize data quality for LLM training.

Location: On-site in San Francisco, London, or New York

Company

A research lab dedicated to making intelligence open and accessible by building foundational AI models.

What you will do

• Build, mentor, and grow a high-performing team of data ingestion engineers.
• Provide leadership across the full ingestion stack including web crawling, extraction pipelines, and data lakes.
• Manage day-to-day execution and prioritize work in a fast-paced environment.
• Guide architectural decisions focusing on scalability, reliability, and auditability.
• Partner with research and data quality teams to measure and improve downstream model impact.
• Collaborate with legal and partnerships teams to onboard new data sources.

Requirements

• Experience building and leading data or infrastructure engineering teams.
• Deep expertise in web-scale data acquisition or ingestion systems at multi-TB to PB scale.
• Strong coding ability and technical credibility.
• Deep expertise in at least one of: web crawling, large-scale extraction, or data lake storage.
• Fluency with modern data tools like Ray, Beam, Spark, Airflow, and object-store architectures.
• Intuition for LLM training data requirements and experimental design.

Culture & Benefits

• Top-tier compensation including salary and stock options.
• Comprehensive medical, dental, and vision insurance.
• 22 weeks of paid parental leave for all parents.
• Unlimited PTO in the U.S. and 30 days in the U.K.
• Visa sponsorship support for exceptional talent.
• Daily provided meals and regular team off-sites.