Назад
Company hidden
обновлено 15 дней назад

AI Evaluation Systems Engineering Lead (AI)

142 800 - 274 800$
Формат работы
onsite
Тип работы
fulltime
Грейд
lead
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
AI Evaluation Systems Engineering Lead (AI): Building evaluation infrastructure that converts model checkpoints and deployed models into reliable, reproducible results with an accent on benchmark configuration, inference, grading, aggregation, and large-scale execution. Focus on improving GPU efficiency and evaluation throughput, diagnosing distributed-system failures, ensuring operational reliability, and leading the engineering team through complex infrastructure delivery.

Location: Mountain View, California, United States

Base pay: USD $142,800–$274,800 per year across the U.S.; USD $188,000–$304,200 per year in the San Francisco Bay Area and New York City metropolitan area for IC5. IC6 base pay: USD $165,600–$296,400 per year across the U.S.; USD $220,800–$331,200 per year in those metropolitan areas.

Company

hirify.global AI develops frontier AI models and the infrastructure used to evaluate, improve, and release them reliably.

What you will do

  • Set the Eval Systems team’s architecture, technical roadmap, delivery priorities, and long-term reliability strategy.
  • Build end-to-end evaluation infrastructure for benchmark configuration, model inference, grading, aggregation, and results publication.
  • Accelerate research iteration by improving evaluation throughput, scheduling, turnaround time, and GPU efficiency.
  • Establish reproducibility, traceability, validation, regression checks, observability, failure recovery, and on-call practices.
  • Contribute code, lead design reviews, and debug problems across evaluation, inference, and distributed execution.
  • Hire, mentor, and develop engineers while aligning research and engineering partners on priorities, dependencies, and tradeoffs.

Requirements

  • Bachelor’s degree in Computer Science or a related technical discipline and 6+ years of technical engineering experience with coding in C, C++, C#, Java, JavaScript, or Python, or equivalent experience.

Nice to have

  • Bachelor’s degree in Computer Science or a related technical field and 10+ years of technical engineering experience.
  • Experience building and operating production distributed systems, infrastructure platforms, or ML systems.
  • Technical leadership experience setting direction and delivering complex projects through a team.
  • Strong Python programming skills and experience contributing to large shared codebases.
  • Experience diagnosing distributed-system failures and improving reliability, performance, and resource efficiency.
  • Production operations experience, including monitoring, incident response, and on-call practices.
  • Experience translating evolving requirements into engineering priorities and partnering with research and engineering teams.
  • Experience mentoring engineers, hiring people, managing teams, or strengthening engineering practices.
  • Experience with LLM evaluation, benchmarking, model-based grading, reinforcement learning infrastructure, GPU-backed inference, distributed training, or large-scale compute workloads.
  • Familiarity with Ray, Kubernetes, Slurm, or other distributed execution frameworks and schedulers.
  • Experience improving experimental reproducibility, validating metrics, or investigating differences between evaluation and serving environments.
  • Experience simplifying complex infrastructure and reducing operational burden through automation and systematic reliability improvements.

Culture & Benefits

  • Hands-on engineering leadership role combining architecture, implementation, team development, and cross-functional collaboration.
  • Focus on clear ownership, collaboration, technical excellence, inclusive teamwork, and dependable operational practices.
  • Benefits and other compensation may be available.
  • The position remains open for at least 5 days, with applications accepted on an ongoing basis until filled.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →