Infrastructure Engineer (GPU Cluster Infrastructure)

Location: Remote in most countries or in-person in Berkeley, California, or Singapore. Visa sponsorship is available for in-person employees.

Compensation: $150,000–$275,000 per year, adjusted for location and experience.

Company

hirify.global is a nonprofit AI research institute focused on making advanced AI safe and beneficial for everyone.

What you will do

• Operate and scale Kubernetes GPU clusters, including node lifecycle, upgrades, driver and image rollouts, rollback procedures, and capacity planning.
• Own batch scheduling and multi-tenancy through queues, quotas, priorities, preemption, gang scheduling, and fair share.
• Design and manage distributed storage, object storage tiers, quotas, backups, and checkpoint I/O.
• Improve fault tolerance for multi-node training by diagnosing NCCL, fabric, GPU, and node-health issues.
• Harden cluster security through identity and access management, network policy, secrets, workload isolation, and sandboxing.
• Work with research teams and infrastructure providers to add capacity, operate clusters, maintain runbooks, and conduct postmortems.

Requirements

• 3+ years of systems or infrastructure engineering experience on production Linux, GPU, HPC, or large-scale batch platforms.
• Production Kubernetes experience for GPU workloads with a batch layer such as Slurm, Kueue, or Volcano.
• Experience with infrastructure as code and observability using tools such as Terraform, Ansible, Helm, ArgoCD, and Prometheus.
• Strong programming ability in Python, Go, Rust, C++, or a comparable infrastructure language.
• Experience owning at least one production system from design through operation and communicating clearly through design documents, incident summaries, and escalations.
• Availability for full-time work, 40 hours per week, with working-hours overlap with Berkeley.

Nice to have

• Distributed training infrastructure using PyTorch, NCCL, NVIDIA drivers, GPU Operator, DCGM, InfiniBand, or RoCE.
• Distributed storage systems such as VAST, Weka, Lustre, Ceph, or large-scale object storage.
• Cluster security, admission control, RBAC, node and container hardening, or sandboxed runtimes.
• Kubernetes scheduler internals, custom controllers, gang scheduling, fair-share, and quota management.
• Experience operating multi-provider platforms across clusters with different network and data-locality characteristics.

Culture & Benefits

• Work directly with AI researchers and engineers on frontier AI safety research infrastructure.
• Remote work is available in most countries; in-person work is available in Berkeley or Singapore.
• Health insurance, paid leave, retirement benefits, and other benefits vary by employment location and arrangement.
• Work computer and work-from-home stipend are provided for eligible employees.
• Travel and equipment expenses are covered; Berkeley-based staff receive catered lunches and dinners.
• Participation in incident coverage and a future on-call rotation is expected as the infrastructure grows.