AI Infrastructure Engineer (LLM Serving)

TL;DR

AI Infrastructure Engineer (LLM Serving): Designing and building platforms for scalable, reliable, and efficient serving of LLMs with an accent on backend system design and infrastructure scalability. Focus on building fault-tolerant systems, optimizing model routing, and implementing observability for large-scale AI production environments.

Location: London, UK

Company

Scale AI provides high-quality data and full-stack technologies that power the world's leading AI models for enterprises and governments.

What you will do

• Build and maintain fault-tolerant, high-performance systems for serving LLMs and other models at scale.
• Develop an internal platform to empower LLM capability discovery.
• Collaborate with researchers and engineers to integrate and optimize models for production and research use cases.
• Conduct architecture and design reviews to ensure best practices in system design and scalability.
• Develop monitoring and observability solutions to ensure system health and performance.
• Lead end-to-end projects from requirements gathering to implementation in a cross-functional environment.

Requirements

• 4+ years of experience building large-scale, high-performance backend systems.
• Strong programming skills in Python, Go, Rust, or C++.
• Experience with LLM serving and routing fundamentals such as rate limiting, token streaming, and load balancing.
• Understanding of LLM concepts including reasoning, tool calling, and prompt templates.
• Experience with containers and orchestration tools like Docker and Kubernetes.
• Familiarity with cloud infrastructure (AWS, GCP) and Infrastructure as Code (Terraform).

Nice to have

• Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference.

Culture & Benefits

• Inclusive and equal opportunity workplace committed to diversity.
• Collaborative environment bridging the gap between AI research and engineering.
• Provision of reasonable accommodations for applicants with physical and mental disabilities.