Staff Engineer Distributed Storage HPC and AI Infrastructure

Staff Engineer Distributed Storage HPC and AI Infrastructure

Company

Together AI

Conditions

6 days ago

Skills

Candidate Availability

Required and preferred rules are kept separate and reflect the wording in the original posting.

About the Role

You will design and deliver multi-petabyte storage systems for AI training and inference workloads. You will architect parallel filesystems and object stores, integrate storage technologies, optimize costs and data paths, and build Kubernetes-native storage operators. You will improve reliability through monitoring, disaster recovery, automation, and performance tuning while partnering with ML and SRE stakeholders.

Requirements

• 8+ years of storage engineering experience, including 3+ years managing distributed storage at multi-petabyte scale.
• Experience deploying and operating high-performance storage for GPU or HPC clusters.
• Production experience with Kubernetes and cloud-native storage.
• Strong Go and Python programming skills for production tools.
• Technical leadership experience improving performance, reliability, or cost efficiency.
• Expertise with distributed storage systems such as WekaFS, Lustre, GPFS, or BeeGFS.
• Production object-storage experience with S3, MinIO, Ceph, or R2.
• Knowledge of Kubernetes CSI drivers, StatefulSets, PersistentVolumes, storage operators, and custom controllers.
• Experience with GPU storage optimization, RDMA or InfiniBand, and parallel filesystem optimization.
• Experience with Terraform, Ansible, Helm, GitOps, and ArgoCD.
• Advanced Linux storage knowledge, including filesystems, LVM, NVMe, and RAID.
• Experience with Prometheus, Grafana, and Thanos.

Responsibilities

• Design multi-petabyte AI and ML storage systems and optimize capacity and costs.
• Design and optimize RDMA, InfiniBand, 400GbE, NVMe-oF, iSCSI, and TCP/IP storage networking.
• Build Kubernetes storage operators and controllers for automated provisioning, multi-tenancy, and quotas.
• Optimize caching, parallel filesystems, and data paths for GPU nodes at scale.
• Build multi-tier caches and implement data-locality, model-weight distribution, prefetching, and eviction strategies.
• Implement monitoring, alerting, SLOs, disaster recovery, backups, runbooks, and chaos engineering.
• Partner with ML and SRE teams, mentor on storage practices, contribute to open source, and write documentation and postmortems.

Benefits

• Hybrid working with two days per week at the Amsterdam office.

Показать контакты