AI Engineer - Data and Evals (AI)

Location: Hybrid, 4 days per week in the Mountain View or San Francisco office

Base salary: $200,000–$280,000 annually

Company

Glean provides an enterprise Work AI platform with search, an AI assistant, scalable AI agents, SaaS connectors, LLM integrations, enterprise graphs, and APIs.

What you will do

• Build evaluation datasets and benchmarks from expert-curated, production, synthetic, and historical failure data, covering edge cases and key user segments.
• Scale evaluation platforms and agent observability through experiment pipelines, trace capture and replay, dashboards, and reproducible cost-efficient runs.
• Develop agentic LLM verifiers for correctness, groundedness, citation quality, and task success, calibrating them against human labels and clear rubrics.
• Run evaluations across CI, pre-release suites, and sampled production traffic, reporting results with confidence intervals.
• Build enterprise simulation environments with simulated users and realistic company data for testing multi-step agent tasks.
• Diagnose failures by slice and trace and turn evaluation results into fixes while balancing quality, latency, and cost; partner with R&D, product, and GTM.

Requirements

• 5+ years of software engineering experience with strong Go and/or Python skills and backend fundamentals including APIs, data modeling, testing, and distributed pipelines.
• Experience building datasets and benchmarks for LLM use cases, RAG, or agents, including evaluation-set curation and synthetic data generation.
• Familiarity with deterministic metrics, LLM-as-a-judge, pointwise and pairwise grading, human labeling, and judge calibration.
• Experience building evaluation or experiment infrastructure that runs reliably at scale, including CI/CD integrations and observability tooling such as OpenTelemetry.
• Strong analytical skills, including investigating traces and failure cases, writing SQL, and using data and metrics to explain quality changes.

Nice to have

• TypeScript experience.
• Experience with simulation environments and/or model post-training techniques.

Culture & Benefits

• Hybrid work with four days per week in the Mountain View or San Francisco office.
• Medical, vision, and dental coverage, a generous time-off policy, and a 401(k) contribution plan.
• Home office improvement, annual education, and wellness stipends.
• Regular company events and daily healthy lunches.
• Inclusive workplace committed to diverse backgrounds, experiences, and perspectives.

Hiring process

• The interview process includes a brief AI-focused exercise or discussion about designing and using AI.