Site Reliability Engineer - Fleet Command & Control (AI)

Location: On-site in San Jose, United States (Santana Row). Relocation support is available for candidates moving to San Jose. The role participates in a follow-the-sun on-call rotation with SREs in other time zones.

Company

hirify.global builds chips, racks, software, and manufacturing systems for frontier AI inference, with a focus on high throughput and low latency.

What you will do

• Define healthy-state signals, SLOs, alerting standards, and automated actions for inference sites, racks, nodes, and chips.
• Reduce alert noise by suppressing expected events, correlating incidents, and retiring ineffective pages.
• Build guarded agentic recovery workflows for restarts, node drains, traffic shifts, spare swaps, and rollbacks.
• Route incidents to domain SREs and coordinate on-site technicians for tray swaps, reseats, power issues, and leak response.
• Shape fleet telemetry and OpenTelemetry tracing, report reliability trends, and improve canary and rolling deployment practices.
• Lead incident management, postmortems, corrective actions, on-call rotations, handoffs, and onboarding.

Requirements

• 2+ years of experience in site reliability, production engineering, or infrastructure for large-scale production systems; 8+ years for the San Jose lead seat.
• Experience defining alerting standards adopted by multiple teams using Prometheus, Grafana, or similar tools; strong knowledge of SLOs, error budgets, and alert design.
• Production on-call experience for externally used services, including incident leadership and blameless postmortems.
• Experience debugging unfamiliar production systems under time pressure using logs, metrics, traces, and core dumps.
• Experience with application or scripting languages such as Python, Golang, or Rust.
• Hands-on experience with Linux, containers, Kubernetes, and infrastructure as code such as Terraform or Ansible.
• Fluency with Linux debugging tools including perf, strace, gdb, dmesg, journalctl, lspci, and tcpdump.
• Comfort troubleshooting hardware-related failures including PCIe device loss, ECC errors, power events, and thermal events.
• Judgment to determine when automation should act independently and when it should page a person, plus clear writing and the ability to align five or more teams on shared standards.

Nice to have

• Experience operating GPU, accelerator, or HPC fleets.
• Experience with LLM inference serving, including prefill, decode, KV cache, batching, and time-to-first-token metrics.
• Network debugging across switches, load balancers, Cloudflare, and edge traffic controls.
• Experience with BMC, Redfish, IPMI, fleet-wide firmware rollouts, data-center telemetry, power distribution, liquid cooling, CDUs, or DCIM.
• Experience with PagerDuty or similar incident tooling, colocation providers, remote-hands technicians, or scaling an SRE function at a startup.

Culture & Benefits

• Fully in-person work in San Jose, with engineering and research responsibilities spanning disciplines.
• Medical, dental, and vision coverage with generous premium support, plus a $500 monthly credit for waiving medical benefits.
• $2,500 monthly housing subsidy for employees living within walking distance of the office.
• Relocation support for moves to San Jose, wellness benefits, and daily office lunch and dinner.
• Unlimited compute budget subject to ROI justification.