Inference Performance Engineer (AI)

Location: San Francisco; hybrid work with in-person collaboration in the Bay Area

Company

Builds AI systems designed to adapt in real time through efficient, flexible, personalized, and accessible intelligence.

What you will do

• Own the cost and performance of the model inference stack as workloads, traffic, and hardware change.
• Improve throughput, cost, and tail latency through KV-cache management, continuous batching, speculative decoding, and quantization.
• Optimize long-context prefill and decode workloads using real production traffic.
• Tune routing between internal infrastructure and external providers based on cost, capacity, and performance.
• Optimize serving engines such as vLLM, SGLang, and TensorRT-LLM, including below-framework work when needed.
• Build profiling and measurement systems to identify time, memory, and compute usage.

Requirements

• 5+ years of experience in ML systems, inference infrastructure, or performance engineering, with measurable cost or latency improvements.
• Deep understanding of model serving, including prefill and decode, memory bandwidth, batching, and concurrency.
• Production experience with vLLM, SGLang, or TensorRT-LLM.
• Strong Python skills and proficiency in C++, Rust, or another systems language.
• Experience with GPU performance, including CUDA, NCCL, mixed precision, memory layout, kernels, or quantization.

Culture & Benefits

• In-person collaboration in the Bay Area within a distributed, global-first team.
• Team offsites and an annual travel stipend for exploring a new country.
• Weekly meal allowance for takeout or grocery delivery.
• Comprehensive medical benefits and generous paid time off.
• Collaborative environment encouraging adaptable teammates and bold ideas.