Назад
Company hidden
7 дней назад

Member of Technical Staff, Pretraining Evaluations (AI)

180 000 - 450 000$
Формат работы
onsite
Тип работы
fulltime
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Member of Technical Staff, Pretraining Evaluations (AI): Building reproducible evaluation systems for multimodal foundation models across text, code, math, reasoning, audio, speech, image, and video with an accent on statistical rigor, benchmark quality, and checkpoint comparison. Focus on designing evaluation harnesses, quantifying uncertainty, detecting data contamination, and developing benchmarks that guide training recipes and capability improvements.

Location: San Jose, United States

US base salary: $180,000–$450,000 annually

Company

hirify.global is an artificial intelligence company developing personalized, proactive, multimodal intelligence and next-generation hardware that interacts with people and the real world.

What you will do

  • Build and maintain evaluation harnesses for held-out loss, domain performance, and capabilities across text, code, math, and reasoning.
  • Design evaluations for early checkpoints and smaller models using log-likelihood scoring, few-shot prompts, and generated completions.
  • Assess benchmark coverage and saturation, quantify uncertainty, and identify training-data contamination with data researchers.
  • Version datasets, prompts, graders, and inference settings to make comparisons reproducible across checkpoints and training runs.
  • Analyze experiments with researchers to improve training recipes, select checkpoints, and prioritize capability improvements.
  • Develop new benchmarks and extend evaluation tools to image, video, speech, and audio tasks.

Requirements

  • Experience building model evaluation systems or experimental tools for reproducible comparisons across models or checkpoints.
  • Knowledge of language model evaluation methods, including log-likelihood and generative scoring, prompt sensitivity, few-shot variance, and formatting effects.
  • Strong statistical judgment in uncertainty estimation, sample-size planning, and distinguishing statistical significance from practical improvements.
  • Experience identifying benchmark contamination, saturation, or gaps between benchmark scores and intended capabilities.
  • Strong Python engineering skills and experience building reliable, efficient evaluation pipelines across many checkpoints.
  • A record of developing useful benchmarks, evaluation methods, or tools that inform research and engineering decisions.

Nice to have

  • Experience designing benchmarks with human ratings or model-based graders.
  • Experience studying capability scaling or predicting larger-model performance from smaller experiments.
  • Experience with code execution sandboxes, math verifiers, or other programmatic graders.
  • Experience evaluating checkpoints during pretraining or mid-training.
  • Multimodal evaluation experience with audio, speech, vision, or interleaved inputs.
  • Experience evaluating model quality under constrained inference budgets.

Culture & Benefits

  • Full-time position with an annual US base salary range of $180,000–$450,000.
  • Total compensation may include additional components and benefits depending on the role.
  • Additional compensation details are shared if an employment offer is extended.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →