Вакансия из Hirify Global, списка международных tech-компаний Для мэтча и отклика нужен Plus
Пожаловаться
Зарплата и рынок
ЗП не указанаразвернуть
На похожих ролях по рынку
$218к$248кмедиана$285к
50 в выборке
На СНГ рынке: $4.3к/мес ($3.6к - $5.1к)
91
Крутая вакансия
развернуть
Понятная зона ответственности, конкретные критерии успеха и влияние на выпускаемые AI-модели делают роль значимой; при этом небольшая команда означает высокий уровень личной ответственности.
?Широкая ответственность
Работа на переднем крае AI
Актуальный стек
Публикации и open source
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Подключить Plus
Описание вакансии
Текст:
Hirify AI/Оригинал
TL;DR
Member of Technical Staff - Evaluations (AI): Building and operating reliable, reproducible internal evaluation infrastructure for post-training decisions, with an accent on held-out benchmarks, contamination checks, and small edge-model evaluation. Focus on designing hard-to-game capability evaluations, debugging generated outputs and inference failures, and releasing open-source evals with research papers and failure-mode reports.
Location: San Francisco, hybrid
Company
hirify.global, spun out of MIT CSAIL, builds general-purpose AI systems optimized for low latency, minimal memory usage, privacy, and reliability across data-center and on-device hardware.
What you will do
Build, maintain, optimize, and scale the internal evaluation suite and infrastructure for post-training decisions.
Design held-out evaluations for agentic workflows, computer use, cybersecurity, and other priority capabilities, including documented construction and contamination checks.
Develop evaluations for small edge models and debug unexpected scores by reviewing individual generated outputs.
Turn customer and open-source feedback into environments for data, training, and evaluation, and partner with audio and other teams on model-specific evaluations.
Release open-source evaluations, write papers, and publish failure-mode reviews of popular evaluations.
Requirements
Strong evaluation-design ability: translate high-level capabilities into accurate, general-purpose evaluations that distinguish model quality and are difficult to game.
Recognizable or verifiable evaluations built previously, with demonstrated coverage, reproducibility, contamination handling, and useful signal.
At least one evaluation built from scratch, such as a benchmark, evaluation paper, or substantial contribution to lm-eval-harness, lighteval, OLMES, HELM, or Inspect.
Fluency in contamination, saturation, judge bias, and variance.
Hands-on experience running and managing evaluations in an evaluation, training, engineering, or DevOps team.
Experience debugging unexpected scores by reading individual generated answers before relying on aggregate results.
Strong software engineering skills sufficient to own evaluation infrastructure and debug inference and infrastructure failures at scale independently.
Proactive execution, parallelizing work, making progress visible, and completing work without close supervision.
Willingness to identify errors in personal evaluations or scores and present results with their caveats.
Nice to have
Model training experience.
Publications or a research background; industry experience is also welcome.
Deep systems expertise.
Familiarity with the capabilities and limitations of small edge models.
Training-team or DevOps experience involving substantial evaluation work; an evaluation-specific title is not required.
Culture & Benefits
The role owns the full evaluation function alongside one engineer, with compute for evaluations at scale and support for open-source and publication work.
Competitive base salary plus equity; final offers reflect skills, experience, and location.
Unlimited PTO, company-wide Refill Days, and shared long weekends.
Comprehensive health coverage, with details varying by location.
Paid leave for pregnancy, parenting, medical needs, and family caregiving.
Будьте осторожны
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →