Назад
8 дней назад

Site Reliability Engineer (AI Infrastructure)

Формат работы
onsite
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Site Reliability Engineer (AI Infrastructure): Designing monitoring, incident response, and reliability systems for a large-scale AI training campus with an accent on GPU fleets, facility telemetry, and cross-discipline infrastructure. Focus on leading SEV incidents, controlling high-risk power and cooling changes, and improving reliability across compute, network, storage, and facility systems.

Location: Southaven, Mississippi or Memphis, Tennessee, United States. On-site participation in the Memphis / Southaven data center campus and its on-call rotation is required.

Company

Building a full super-intelligence stack spanning large-scale compute campuses, AI models, and user-facing applications.

What you will do

  • Design monitoring architecture and improve signal quality across GPU, host, network, storage, power, cooling, and facility telemetry.
  • Lead technical response to SEV-class incidents, coordinating incident bridges, NOC escalation, paging, timelines, and severity management.
  • Run blameless postmortems and drive corrective actions through closure.
  • Gate high-risk firmware, BIOS, BMC, power, and cooling changes using canary, rollback, and formal change-management processes.
  • Lead cross-functional reliability projects addressing failures across compute, network, storage, and facility boundaries.
  • Build playbooks, runbooks, dependency maps, game days, error budgets, and availability objectives with NOC and infrastructure partners.

Requirements

  • Bachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or a related field, or equivalent experience.
  • At least 5 years of experience in site reliability, systems engineering, plant operations, or large-scale production operations.
  • Experience leading large-scale incidents and providing calm technical leadership from an SRE/NOC bridge or plant control room.
  • Fleet- or campus-scale monitoring and observability experience, including alert hygiene, suppression, and signal quality.
  • Hands-on experience with Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls such as SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus, or BACnet.
  • Experience across at least two of compute, network, storage, power, and cooling/facilities telemetry, plus scripting with Python or Bash and experience with a systems language.

Nice to have

  • Power plant, nuclear plant, high-consequence industrial control, data center, or similar facilities experience.
  • AI/ML infrastructure or supercomputing experience with GPU fleets, NCCL, InfiniBand/RoCE, Slurm, or Kubernetes.
  • Firmware and BMC management, including Redfish, IPMI, staged rollouts, and release validation.
  • Reliability engineering experience with MTBF/AFR analysis, grey-failure diagnosis, reliability-centered maintenance, and vendor RCA/RMA programs.
  • Experience with CMMS, commissioning, integrated systems testing, SLOs, SLIs, error budgets, game days, or dependency mapping.

Culture & Benefits

  • Work on one of the largest AI training campuses and its super-intelligence infrastructure.
  • Collaborate with NOC, data center operations, hardware failure analysis, and infrastructure engineering.
  • Participate in 24/7 operations and incident-response practices.
  • Use blameless postmortems and closed-loop corrective action to improve system reliability.

Будьте осторожны: если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого - это мошенники. Обязательно жмите "Пожаловаться" или пишите в поддержку. Подробнее в гайде →