Senior Network Production Engineer, AI Supercomputing

Location: London, United Kingdom

Salary: USD $119,800–$234,700 per year for Software Engineering IC4 roles; other ranges apply to IC5 and specific U.S. locations.

Company

hirify.global AI builds and operates frontier-scale AI supercomputers used to train advanced machine-learning models.

What you will do

• Own the availability, performance, operational readiness, and health indicators of the MRC Ethernet backend network.
• Respond to high-severity incidents and diagnose failures across GPUs, NICs, switches, cables, firmware, drivers, operating systems, and distributed workloads.
• Develop automated mitigation for path avoidance, node quarantine, workload relocation, component remediation, and fleet-wide changes.
• Improve reliability and performance for large distributed training jobs by analyzing collective performance, stalls, timeouts, restarts, and MFU loss.
• Build network qualification gates, failure-injection tests, monitoring, diagnostics, inventory, topology, and configuration automation.
• Lead cross-functional investigations, produce root-cause analyses, and create operational runbooks for global coverage.

Requirements

• Bachelor’s degree in Computer Science or a related technical discipline with 4+ years of experience in SRE, DevOps, or infrastructure engineering, or equivalent experience.
• Experience operating large-scale datacenter or high-performance computing networks in production.
• Experience with Ethernet, RDMA, RoCEv2, congestion control, routing, load balancing, and lossless or near-lossless network design.
• Experience diagnosing failures across switches, NICs, hosts, drivers, firmware, and distributed applications.
• Linux experience and programming skills in Python, Go, C++, or another systems-oriented language.
• Experience building production infrastructure monitoring, automation, diagnostic, or remediation systems and leading high-severity incident response.

Nice to have

• Experience operating GPU training clusters or tightly coupled distributed-computing systems.
• Knowledge of MRC, multi-rail or multipath Ethernet transports, NCCL, CUDA, GPUDirect RDMA, and distributed training frameworks.
• Familiarity with ConnectX NICs, Ethernet switch ASICs, SONiC, SAI, InfiniBand, Kubernetes, Slurm, and cluster-health certification.
• Experience with streaming telemetry, gNMI, Prometheus, Datadog, Kusto, eBPF, or equivalent observability systems.
• Experience managing fleet-wide firmware, driver, or network operating-system changes.

Culture & Benefits

• Participation in a global on-call rotation supporting round-the-clock network operations.
• Collaboration with infrastructure teams, datacenter operators, Azure Networking, NVIDIA, switch and NIC teams, and network-software partners.
• Benefits and additional compensation may be available depending on role and location.