Вакансия из Hirify Global, списка международных tech-компаний Для мэтча и отклика нужен Plus
Пожаловаться
Зарплата и рынок
развернуть
$180к-450к/год
$270к$310кмедиана$320к
По похожим ролям (49) ~ $310к/год
На СНГ рынке: $4.8к/мес ($3.7к - $7.5к)
88
Крутая вакансия
развернуть
Чётко очерченная роль в важном направлении ИИ: вы будете отвечать за оценочные системы, которые влияют на решения по обучению моделей.
Чёткая зона ответственности
Актуальный стек
Передовое направление
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Подключить Plus
Описание вакансии
Текст:
Hirify AI/Оригинал
TL;DR
Member of Technical Staff, Pretraining Evaluations (AI): Building reproducible evaluation systems for multimodal foundation models across text, code, math, reasoning, audio, speech, image, and video with an accent on statistical rigor, benchmark quality, and checkpoint comparison. Focus on designing evaluation harnesses, quantifying uncertainty, detecting data contamination, and developing benchmarks that guide training recipes and capability improvements.
Location: San Jose, United States
US base salary: $180,000–$450,000 annually
Company
hirify.global is an artificial intelligence company developing personalized, proactive, multimodal intelligence and next-generation hardware that interacts with people and the real world.
What you will do
Build and maintain evaluation harnesses for held-out loss, domain performance, and capabilities across text, code, math, and reasoning.
Design evaluations for early checkpoints and smaller models using log-likelihood scoring, few-shot prompts, and generated completions.
Assess benchmark coverage and saturation, quantify uncertainty, and identify training-data contamination with data researchers.
Version datasets, prompts, graders, and inference settings to make comparisons reproducible across checkpoints and training runs.
Analyze experiments with researchers to improve training recipes, select checkpoints, and prioritize capability improvements.
Develop new benchmarks and extend evaluation tools to image, video, speech, and audio tasks.
Requirements
Experience building model evaluation systems or experimental tools for reproducible comparisons across models or checkpoints.
Knowledge of language model evaluation methods, including log-likelihood and generative scoring, prompt sensitivity, few-shot variance, and formatting effects.
Strong statistical judgment in uncertainty estimation, sample-size planning, and distinguishing statistical significance from practical improvements.
Experience identifying benchmark contamination, saturation, or gaps between benchmark scores and intended capabilities.
Strong Python engineering skills and experience building reliable, efficient evaluation pipelines across many checkpoints.
A record of developing useful benchmarks, evaluation methods, or tools that inform research and engineering decisions.
Nice to have
Experience designing benchmarks with human ratings or model-based graders.
Experience studying capability scaling or predicting larger-model performance from smaller experiments.
Experience with code execution sandboxes, math verifiers, or other programmatic graders.
Experience evaluating checkpoints during pretraining or mid-training.
Multimodal evaluation experience with audio, speech, vision, or interleaved inputs.
Experience evaluating model quality under constrained inference budgets.
Culture & Benefits
Full-time position with an annual US base salary range of $180,000–$450,000.
Total compensation may include additional components and benefits depending on the role.
Additional compensation details are shared if an employment offer is extended.
Будьте осторожны
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →