Назад
Company hidden
9 дней назад

Member of Technical Staff - Evaluations (AI)

Формат работы
hybrid
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Member of Technical Staff - Evaluations (AI): Building and operating reliable, reproducible internal evaluation infrastructure for post-training decisions, with an accent on held-out benchmarks, contamination checks, and small edge-model evaluation. Focus on designing hard-to-game capability evaluations, debugging generated outputs and inference failures, and releasing open-source evals with research papers and failure-mode reports.

Location: San Francisco, hybrid

Company

hirify.global, spun out of MIT CSAIL, builds general-purpose AI systems optimized for low latency, minimal memory usage, privacy, and reliability across data-center and on-device hardware.

What you will do

  • Build, maintain, optimize, and scale the internal evaluation suite and infrastructure for post-training decisions.
  • Design held-out evaluations for agentic workflows, computer use, cybersecurity, and other priority capabilities, including documented construction and contamination checks.
  • Develop evaluations for small edge models and debug unexpected scores by reviewing individual generated outputs.
  • Turn customer and open-source feedback into environments for data, training, and evaluation, and partner with audio and other teams on model-specific evaluations.
  • Release open-source evaluations, write papers, and publish failure-mode reviews of popular evaluations.

Requirements

  • Strong evaluation-design ability: translate high-level capabilities into accurate, general-purpose evaluations that distinguish model quality and are difficult to game.
  • Recognizable or verifiable evaluations built previously, with demonstrated coverage, reproducibility, contamination handling, and useful signal.
  • At least one evaluation built from scratch, such as a benchmark, evaluation paper, or substantial contribution to lm-eval-harness, lighteval, OLMES, HELM, or Inspect.
  • Fluency in contamination, saturation, judge bias, and variance.
  • Hands-on experience running and managing evaluations in an evaluation, training, engineering, or DevOps team.
  • Experience debugging unexpected scores by reading individual generated answers before relying on aggregate results.
  • Strong software engineering skills sufficient to own evaluation infrastructure and debug inference and infrastructure failures at scale independently.
  • Proactive execution, parallelizing work, making progress visible, and completing work without close supervision.
  • Willingness to identify errors in personal evaluations or scores and present results with their caveats.

Nice to have

  • Model training experience.
  • Publications or a research background; industry experience is also welcome.
  • Deep systems expertise.
  • Familiarity with the capabilities and limitations of small edge models.
  • Training-team or DevOps experience involving substantial evaluation work; an evaluation-specific title is not required.

Culture & Benefits

  • The role owns the full evaluation function alongside one engineer, with compute for evaluations at scale and support for open-source and publication work.
  • Competitive base salary plus equity; final offers reflect skills, experience, and location.
  • Unlimited PTO, company-wide Refill Days, and shared long weekends.
  • Comprehensive health coverage, with details varying by location.
  • Paid leave for pregnancy, parenting, medical needs, and family caregiving.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →