Research Infrastructure Engineer (AI)

Location: Remote and globally executable; optional office work in London, New York, San Francisco, or Warsaw. The role is also listed for the United States, Bulgaria, Poland, and the United Kingdom.

Company

ElevenLabs is an AI research and product company developing voice, creative media, conversational agent, and AI audio platforms.

What you will do

• Operate and improve the NVIDIA GPU fleet through provisioning, scheduling, monitoring, upgrades, and capacity planning.
• Build automation for node health checks, draining, remediation, burn-in pipelines, and other fleet-health workflows.
• Own the infrastructure stack beneath training workloads, including OS images, NVIDIA drivers, CUDA, container runtimes, NCCL, InfiniBand, and RoCE.
• Run and tune Slurm or a similar scheduler, and build and maintain high-performance storage for datasets and checkpoints.
• Diagnose and resolve systemic performance issues involving stragglers, degraded links, thermal problems, and unreliable GPUs.
• Evaluate rented GPU capacity, support hardware deployment, and maintain secure access control, network isolation, and secrets management.

Requirements

• Production experience running large-scale Linux server or GPU environments, with willingness to both build and operate them.
• Strong knowledge of the NVIDIA stack, including drivers, CUDA, NCCL, and DCGM, or deep systems experience with the ability to learn hardware stacks quickly.
• Comfort working with bare-metal environments, server hardware, and high-speed networking.
• Ability to write reliable automation in Python and/or Bash using infrastructure-as-code tools such as Ansible or Terraform.
• Ability to investigate metrics, logs, and PromQL data to identify root causes.
• Ownership of technical scope end to end and willingness to handle all required tasks, including datacenter trips.

Nice to have

• Experience supporting ML training workloads from the infrastructure side, including distributed training failure modes and checkpointing patterns.
• Experience evaluating and working with GPU cloud providers.
• Experience with parallel filesystems such as WEKA or VAST, or large-scale object storage.
• Experience with BMC, IPMI, or Redfish automation and PXE provisioning at scale.
• Awareness of power and cooling requirements for dense GPU deployments.

Culture & Benefits

• High-velocity environment with rapid experimentation, lean autonomous teams, and minimal bureaucracy.
• AI-first culture focused on technical excellence and measurable impact.
• Annual discretionary stipend for professional development.
• Annual discretionary stipend for meeting colleagues and company offsites.
• Monthly coworking stipend for people outside the main hubs.