Назад
2 месяца назад

Principal Infrastructure Engineer, AI Cluster Performance & Validation

Формат работы
onsite
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Principal Infrastructure Engineer, AI Cluster Performance & Validation (AI/HPC infrastructure): Building control-plane systems, validation pipelines, and performance tooling for multi-thousand-GPU AI and HPC clusters with an accent on distributed workloads, cluster qualification, and full-stack performance diagnosis. Focus on isolating GPU, networking, storage, scheduler, and communication-library bottlenecks while automating burn-in, telemetry correlation, regression detection, and fleet-wide optimization.

Location: Houston, New York, San Francisco, or Seattle, United States

Company

Nscale operates AI infrastructure and large-scale AI and high-performance computing environments.

What you will do

  • Define architecture, roadmaps, acceptance criteria, and performance standards for multi-thousand-GPU clusters.
  • Run distributed training and inference workloads across large-scale AI clusters to validate production behavior.
  • Lead root-cause analysis of cluster failures and performance regressions across GPUs, networking, storage, schedulers, and communication libraries.
  • Design automated validation, burn-in, benchmarking, thermal, power, and reliability qualification systems for nodes, racks, and full pods.
  • Optimize fabric configuration, collective communication, RDMA and storage paths, GPU settings, NUMA placement, and host performance.
  • Build production-grade Python tooling for automated triage, telemetry correlation, regression detection, and operational workflows.

Requirements

  • Bachelor’s degree in Computer Science, Computer Engineering, or a related technical field, or equivalent practical experience.
  • 10+ years of experience building, operating, or debugging large-scale compute infrastructure, including significant staff- or principal-level technical leadership.
  • Hands-on experience running distributed AI training or inference at scale with parallelism strategies and frameworks such as PyTorch, Megatron-LM, or DeepSpeed.
  • Experience validating and accepting clusters with thousands of GPUs and defending benchmark results to engineers and customers.
  • Deep expertise in Linux systems, Python, high-performance networking, InfiniBand or RoCEv2, RDMA, GPUDirect, NUMA, PCIe, and performance debugging.
  • Experience operating AI workloads under SLURM or Kubernetes at scale, plus experience with C/C++ or Go and configuration management tools such as Ansible or Terraform.

Nice to have

  • Master’s degree or PhD in Engineering, Computer Science, or a related field.
  • Experience bringing up a greenfield GPU supercluster and standardizing firmware, drivers, and topology across a heterogeneous fleet.
  • Experience with NVIDIA GPU platforms, NVLink, NVSwitch, DCGM, SHARP, UFM, or AMD Instinct and ROCm/RCCL.
  • Experience with MLPerf, advanced GPU and fabric observability, parallel storage, cloud-native infrastructure, and bare-metal tooling.
  • Experience using AI tools to improve infrastructure workflows and operator experience.

Culture & Benefits

  • Cross-functional collaboration with Infrastructure, Platform, SRE, and customer-facing teams.
  • Responsibility for engineering standards covering reliability, observability, benchmarking, and operational excellence.
  • Opportunity to mentor engineers, create runbooks, and produce post-incident technical write-ups.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →