Principal Infrastructure Engineer, AI Cluster Performance & Validation

Location: Houston, New York, San Francisco, or Seattle, United States

Company

Nscale operates AI infrastructure and large-scale AI and high-performance computing environments.

What you will do

• Define architecture, roadmaps, acceptance criteria, and performance standards for multi-thousand-GPU clusters.
• Run distributed training and inference workloads across large-scale AI clusters to validate production behavior.
• Lead root-cause analysis of cluster failures and performance regressions across GPUs, networking, storage, schedulers, and communication libraries.
• Design automated validation, burn-in, benchmarking, thermal, power, and reliability qualification systems for nodes, racks, and full pods.
• Optimize fabric configuration, collective communication, RDMA and storage paths, GPU settings, NUMA placement, and host performance.
• Build production-grade Python tooling for automated triage, telemetry correlation, regression detection, and operational workflows.

Requirements

• Bachelor’s degree in Computer Science, Computer Engineering, or a related technical field, or equivalent practical experience.
• 10+ years of experience building, operating, or debugging large-scale compute infrastructure, including significant staff- or principal-level technical leadership.
• Hands-on experience running distributed AI training or inference at scale with parallelism strategies and frameworks such as PyTorch, Megatron-LM, or DeepSpeed.
• Experience validating and accepting clusters with thousands of GPUs and defending benchmark results to engineers and customers.
• Deep expertise in Linux systems, Python, high-performance networking, InfiniBand or RoCEv2, RDMA, GPUDirect, NUMA, PCIe, and performance debugging.
• Experience operating AI workloads under SLURM or Kubernetes at scale, plus experience with C/C++ or Go and configuration management tools such as Ansible or Terraform.

Nice to have

• Master’s degree or PhD in Engineering, Computer Science, or a related field.
• Experience bringing up a greenfield GPU supercluster and standardizing firmware, drivers, and topology across a heterogeneous fleet.
• Experience with NVIDIA GPU platforms, NVLink, NVSwitch, DCGM, SHARP, UFM, or AMD Instinct and ROCm/RCCL.
• Experience with MLPerf, advanced GPU and fabric observability, parallel storage, cloud-native infrastructure, and bare-metal tooling.
• Experience using AI tools to improve infrastructure workflows and operator experience.

Culture & Benefits

• Cross-functional collaboration with Infrastructure, Platform, SRE, and customer-facing teams.
• Responsibility for engineering standards covering reliability, observability, benchmarking, and operational excellence.
• Opportunity to mentor engineers, create runbooks, and produce post-incident technical write-ups.