Назад
Company hidden
23 часа назад

Member of Technical Staff, Evals & Data Quality (AI)

140 000 - 200 000$
Формат работы
hybrid
Тип работы
fulltime
Английский
b2
Страна
US/Canada
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Member of Technical Staff, Evals & Data Quality (AI): Building benchmark methodologies, audit rubrics, automated graders, and quality standards for robotics and embodied AI data with an accent on machine learning evaluation, statistical measurement, and data quality. Focus on designing ground-truth pipelines, detecting metric gaming, reducing manual review costs, and making evidence-based ship or no-ship decisions.

Location: Hybrid in New York, Toronto GTA, or San Francisco

Salary: Toronto: CA$140K–CA$200K; New York and San Francisco: $140K–$200K

Company

hirify.global builds data infrastructure for robotics and embodied AI, including data capture, labeling, and hardware-enabled workflows used to train and validate humanoid and embodied AI systems.

What you will do

  • Own benchmark methodology, including evaluation sets, sampling, metrics, versioning, and comparisons.
  • Write customer quality specifications and audit rubrics aligned with customer rejection criteria.
  • Run human audits and coordinate with annotation and operations on sampling, agreement, throughput, and cost.
  • Build automated checks and graders that reduce manual review costs and resist gaming.
  • Track lost recorded hours, report quality results, and make ship or no-ship decisions on data.

Requirements

  • Experience in evaluation, measurement, or data quality for machine learning systems, or in data operations at scale.
  • Machine learning and data science background with strong Python skills.
  • Working knowledge of statistics, including sampling, precision and recall, agreement, and confidence intervals.
  • Experience designing evaluation protocols and ground-truth pipelines from scratch.
  • Ability to challenge research teams’ results and support challenges with evidence.
  • Ability to identify how metrics can be gamed and close the resulting gaps.

Nice to have

  • Experience with 3D pose, trajectory, and reconstruction metrics such as MPJPE and ATE.
  • Knowledge of sampling design, inter-annotator agreement, loss accounting, and managing annotators or vendors.
  • Experience with verifiers, learned scoring, active learning, and human-in-the-loop evaluation.
  • Experience with quality scoring, filtering, and composition of large datasets.

Culture & Benefits

  • Inclusive work environment welcoming applicants from all backgrounds.
  • Equal-opportunity hiring and accommodations support during recruitment.
  • AI-assisted tools may support application evaluation, with final consideration remaining subject to human judgment.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →