AI Infrastructure Engineer (AI)

TL;DR

AI Infrastructure Engineer (Model Serving): Designing and building platforms for scalable, reliable, and efficient serving of LLMs with an accent on high-performance backend systems and fault tolerance. Focus on optimizing model integration for production, implementing token streaming and load balancing, and ensuring system observability.

Location: Must be based in San Francisco, New York, or Seattle

Salary: $180,000 - $225,000 USD

Company

Scale AI develops reliable AI systems for the world's most important decisions, providing high-quality data and full-stack technologies that power leading models for enterprises and governments.

What you will do

• Build and maintain fault-tolerant, high-performance systems for serving LLM workloads at scale.
• Develop an internal platform to empower LLM capability discovery.
• Collaborate with researchers and engineers to integrate and optimize models for production and research.
• Conduct architecture and design reviews to ensure system scalability and best practices.
• Implement monitoring and observability solutions to ensure system health and performance.
• Lead cross-functional projects end-to-end, from requirements gathering to implementation.

Requirements

• 4+ years of experience building large-scale, high-performance backend systems.
• Strong programming skills in Python, Go, Rust, or C++.
• Experience with LLM serving and routing fundamentals (rate limiting, token streaming, load balancing).
• Understanding of LLM concepts such as reasoning, tool calling, and prompt templates.
• Proficiency with Docker, Kubernetes, and cloud infrastructure (AWS, GCP).
• Experience with infrastructure as code using Terraform.

Nice to have

• Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference.

Culture & Benefits

• Comprehensive health, dental, and vision coverage.
• Retirement benefits and a dedicated learning and development stipend.
• Generous PTO and a commuter stipend.
• Inclusive and equal opportunity workplace culture.