AI Infrastructure Engineer (Serving Platform)

TL;DR

AI Infrastructure Engineer (Serving Platform): Building and optimizing scalable, reliable platforms for the efficient serving of LLMs with an accent on backend system design and ML fundamentals. Focus on implementing fault-tolerant systems, optimizing model routing, and developing internal platforms for LLM capability discovery.

Location: United States (implied by compliance with US Department of Labor)

Company

Scale AI develops reliable AI systems, providing high-quality data and full-stack technologies that power leading models for enterprises and government agencies.

What you will do

• Build and maintain fault-tolerant, high-performance systems for serving LLMs at scale
• Develop an internal platform to facilitate LLM capability discovery
• Collaborate with researchers and engineers to optimize and integrate models for production
• Lead projects end-to-end from requirements gathering to implementation
• Conduct architecture reviews and develop monitoring and observability solutions

Requirements

• 4+ years of experience building large-scale, high-performance backend systems
• Strong programming skills in Python, Go, Rust, or C++
• Experience with LLM serving and routing fundamentals (rate limiting, token streaming, load balancing)
• Knowledge of LLM concepts such as reasoning, tool calling, and prompt templates
• Proficiency with Docker, Kubernetes, and cloud infrastructure (AWS, GCP, Terraform)

Nice to have

• Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference