Machine Learning Infrastructure Engineer

Location

Location: San Francisco, onsite

What you will do

• Own and maintain large-scale model training infrastructure including scheduling, job management, checkpointing, and metrics/logging.
• Scale distributed JAX-based training across TPU and GPU clusters.
• Optimize performance by profiling memory usage, device utilization, throughput, and synchronization.
• Build abstractions for launching, monitoring, debugging, and reproducing experiments.
• Manage cloud-based GPU/TPU compute resources efficiently while controlling costs.
• Collaborate with researchers to translate needs into infrastructure capabilities and best practices.
• Contribute to core JAX training code supporting new architectures and evaluation metrics.

Requirements

• Strong software engineering fundamentals and experience building ML training infrastructure or internal platforms.
• Hands-on experience with large-scale training in JAX (preferred) or PyTorch.
• Familiarity with distributed training, multi-host setups, data loaders, and evaluation pipelines.
• Experience managing training workloads on cloud platforms such as SLURM, Kubernetes, GCP TPU/GKE, or AWS.
• Ability to debug and optimize performance bottlenecks across the training stack.
• Strong cross-functional communication and ownership mindset.

Nice to have

• Deep ML systems background including training compilers, runtime optimization, and custom kernels.
• Experience with GPU/TPU performance tuning and operating close to hardware.
• Background in robotics, multimodal models, or large-scale foundation models.
• Experience designing abstractions balancing researcher flexibility with system reliability.

Culture & Benefits

• Consideration of qualified applicants with arrest and conviction records pursuant to San Francisco Fair Chance Ordinance.