AI Infrastructure Engineer (Model Serving Platform)

TL;DR

AI Infrastructure Engineer (Model Serving Platform): Design and build scalable, reliable, and efficient platforms for serving large language models (LLMs) with an accent on backend system design and ML fundamentals. Focus on building fault-tolerant systems, integrating models for production and research, and ensuring system health and performance.

Location: San Francisco, CA or New York, NY, USA

Salary: $179,400–$224,250 USD

Company

Scale AI develops reliable AI systems powering leading models and applications for enterprises and governments worldwide.

What you will do

• Build and maintain fault-tolerant, high-performance systems for serving LLM workloads at scale.
• Develop an internal platform to enable LLM capability discovery.
• Collaborate with researchers and engineers to integrate and optimize models for production and research.
• Conduct architecture and design reviews to ensure best practices in system design and scalability.
• Develop monitoring and observability solutions to maintain system health and performance.
• Lead projects end-to-end in a cross-functional environment.

Requirements

• Must have 4+ years experience building large-scale backend systems.
• Strong programming skills in Python, Go, Rust, or C++.
• Experience with LLM serving fundamentals and capabilities.
• Familiarity with containers, orchestration (Docker, Kubernetes), and cloud infrastructure (AWS, GCP).
• Ability to work independently in fast-moving environments.
• Location: Must be based in San Francisco or New York, USA.

Nice to have

• Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference.

Culture & Benefits

• Comprehensive health, dental, and vision coverage.
• Retirement benefits and equity compensation.
• Learning and development stipend.
• Generous paid time off.
• Commuter stipend for eligible roles.