Назад
Company hidden
4 дня назад

ML Infra Engineer (AI)

Формат работы
onsite
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
ML Infra Engineer (AI): Building intelligent scheduling, placement, and orchestration systems for large-scale GPU and TPU training clusters with an accent on resource allocation, multi-cluster control planes, and fault-tolerant distributed systems. Focus on optimizing accelerator utilization, implementing priority and preemption policies, and extending compute infrastructure to inference workloads and robot deployment.

Location: San Francisco, United States; Onsite

Company

hirify.global develops general-purpose AI, foundation models, and learning algorithms for robots and physically actuated devices.

What you will do

  • Design and build multi-tenant scheduling and placement systems for training jobs across heterogeneous GPU and TPU clusters.
  • Develop multi-cluster orchestration and control-plane capabilities for job lifecycle management, migration, failover, and rescheduling across on-premises and cloud environments.
  • Optimize accelerator utilization through quota management, priority tiers, preemption, queueing, and fairness policies.
  • Implement fault detection, automatic recovery, health checking, node management, and scaling for long-running multi-node jobs across thousands of accelerators.
  • Extend scheduling and orchestration to inference workloads and model deployment on physical robots.
  • Build dashboards, alerting, SLOs, and debugging tools to improve observability, researcher experience, and cluster reliability.

Requirements

  • Strong software engineering fundamentals.
  • Experience building or operating job scheduling or resource management systems at scale.
  • Experience with large-scale compute clusters using GPUs and/or TPUs.
  • Familiarity with schedulers and orchestration systems such as SLURM, Kubernetes, GKE, K3S, or internal equivalents.
  • Ability to reason about resource allocation, bin-packing, priority scheduling, and multi-tenancy.
  • Understanding of long-running, multi-node ML training workloads, including stragglers and topology dependencies.
  • Willingness to own systems end-to-end, from design through operation, and work closely with researchers to unblock projects.

Nice to have

  • Experience building multi-cluster or federated scheduling systems.
  • Experience with TPU infrastructure, including GCP TPU slices, Multislice, or GKE.
  • Background with Borg, YARN, Mesos, or custom cluster resource managers.
  • Linux systems engineering, networking, and infrastructure-as-code experience.
  • Experience with NCCL, collective communication, and topology-aware placement.
  • Experience with capacity planning and cloud cost optimization at scale.
  • Familiarity with JAX, PyTorch, or similar ML frameworks at the runtime or systems level.

Culture & Benefits

  • Full-time, onsite work in San Francisco.
  • Close collaboration with ML infrastructure, data platform, and research teams.
  • Employment consideration is subject to the San Francisco Fair Chance Ordinance; qualified applicants with arrest and conviction records will be considered.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →