Member of Technical Staff, Evaluations (AI)

Location: New York, NY; San Francisco, or Los Angeles, United States

Salary: $175,000–$250,000 gross annual base salary, plus equity and benefits.

Company

Builds Atlas, an AI platform that applies physics-grounded intelligence to verify, debug, and optimize hardware across its lifecycle.

What you will do

• Own the design, reliability, and automation of evaluation harnesses and recurring reporting pipelines.
• Design ablations to measure the effects of model, harness, tool, data, and context changes on performance, cost, and runtime.
• Analyze agent trajectories, cluster failure modes, and identify root causes such as context limitations, tool inefficiency, and gaps in engineering understanding.
• Develop verifiable scoring systems and calibrate LLM-based judges against expert assessments.
• Partner with Operations and Electrical Engineering to create synthetic datasets covering demanding hardware-engineering tasks.
• Translate customer and partner definitions of quality into measurable evaluation artifacts and communicate results clearly.

Requirements

• Strong Python skills for reliable research or production infrastructure, data pipelines, and tooling.
• Experience designing evaluations, benchmarks, or metrics for ML systems, ideally large language models or agentic systems.
• Hands-on experience with LLM agents, harnesses, tool use, and trajectory analysis.
• Strong knowledge of experimental design and statistics, with sound judgment about trustworthy metrics.
• Excellent written and verbal communication for technical and non-technical audiences.
• Ability to work from one of the listed United States locations and meet applicable U.S. export-control authorization requirements without sponsorship for an export license.

Nice to have

• Electrical or hardware engineering experience, including circuit design, RF/EM, signal integrity, PCB layout, or test and measurement.
• Experience calibrating LLM-as-judge systems, curating synthetic datasets, or evaluating multimodal and spatiotemporal reasoning.
• Experience building trusted dashboards and reporting, or working with customers and external technical partners.

Culture & Benefits

• Full-time role reporting directly to the CTO.
• Medical, vision, and dental insurance with monthly premiums fully covered for employees and dependents.
• 401(k) retirement plan and unlimited PTO.
• Daily lunch provided through local restaurants.
• Relocation support provided.