Member of Technical Staff, Distributed Training Performance (AI)

Location: San Jose, United States

Salary: $180,000–$450,000 annually

Company

hirify.global builds multimodal artificial intelligence systems and next-generation hardware for proactive, personalized interaction through speech, text, vision, and persistent memory.

What you will do

• Own model FLOPs utilization and step time for dense and mixture-of-experts pretraining runs across thousands of GPUs.
• Develop and integrate fused and custom CUDA, Triton, and CUTLASS kernels for attention, normalization, and mixture-of-experts routing.
• Optimize communication, compute overlap, memory use, activation recomputation, sharding, and data, tensor, pipeline, sequence, context, and expert parallelism.
• Profile kernels, communication, data loading, and mixed-precision training to resolve throughput, performance, and numerical stability bottlenecks.
• Build reusable training components for new architectures and multimodal speech, vision, and text encoders.
• Improve checkpointing, restart time, failure detection, observability, and reliability; participate in on-call support during large runs.

Requirements

• Experience developing distributed training systems and working with Megatron-LM, NeMo, or a comparable framework, including parallelism strategies.
• Strong understanding of data, tensor, pipeline, sequence, context, and expert parallelism, including communication patterns, memory tradeoffs, and failure modes.
• Experience using GPU profilers to analyze kernel execution, memory bandwidth, communication overhead, and data-loading stalls.
• Working knowledge of mixed-precision training with BF16 and FP8, including diagnosing numerical instability.
• Strong Python and C++/CUDA engineering skills, with experience building reliable tools for researchers.
• Experience developing CUDA or Triton kernels, or using CUTLASS, for attention, normalization, or mixture-of-experts models.
• A record of measurably improving MFU or throughput on large-scale training jobs.

Nice to have

• Experience with large-scale mixture-of-experts training, expert parallelism, load balancing, and all-to-all communication optimization.
• Experience with NCCL or network topology-aware scheduling.
• Experience with fault-tolerant, asynchronous, in-memory recovery, or elastic training.
• Experience balancing workloads across multimodal encoders.
• Familiarity with Slurm, Kubernetes, and GPU health monitoring at fleet scale.

Culture & Benefits

• Work with architecture researchers and compute infrastructure engineers to align model designs, training software, and cluster capabilities.
• Full-time position with a US base salary range of $180,000–$450,000 annually.
• Total compensation may include additional components and benefits depending on the role.