Principal Model Optimization Engineer (ML)

Location: San Mateo, California, United States; office-based roles are onsite Tuesday, Wednesday, and Thursday, with optional presence on Monday and Friday.

Salary: $295,250–$345,040 USD base pay annually.

Company

Roblox builds tools and platforms for 3D immersive digital experiences created by a global developer and creator community.

What you will do

• Set the technical direction for ML Platform systems supporting hundreds of ML use cases and billions of inferences per day.
• Design platform capabilities adopted by multiple teams, including GPU and compute management, quotas, priority tiers, preemption, multi-cluster scheduling, and capacity planning.
• Define, instrument, and improve metrics such as time from code to first training run, job success rate, GPU utilization, and cost per training or inference run.
• Investigate slow or failing jobs across pipelines, schedulers, GPU profiles, and hardware errors, then implement durable fixes.
• Work with ML engineers, data scientists, Infrastructure, Data Platform, and Security on shared storage, networking, data access, and compliance foundations.
• Raise the technical bar through design reviews, written proposals, code, and mentoring senior engineers.

Requirements

• 6+ years of professional experience and system design experience building performant systems.
• End-to-end experience building ML infrastructure, including training pipelines, workflow orchestration such as Kubeflow, Airflow, or Ray, data or feature pipelines, experiment tracking, model registries, or model serving.
• Experience designing widely adopted SDKs, APIs, or developer tools, with a strong understanding of platform usability.
• Hands-on experience running GPU workloads at scale on Kubernetes, including scheduling, quotas, multi-cluster dispatch, autoscaling, and debugging GPU and node failures such as Xid errors and NCCL hangs.
• Depth in training and inference performance, including distributed training, mixed precision, batching, and quantization.
• Experience setting measurable success metrics, delivering incrementally, and iterating until results improve.
• Experience leading cross-team technical work without direct authority and writing clearly for engineers and leaders.
• Bachelor’s degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.

Culture & Benefits

• Full-time employees are eligible for equity compensation and benefits.
• Roblox provides equal employment opportunities and reasonable accommodations during the recruiting process.
• The role contributes to safer and more civil shared digital experiences.