Engineering Manager (GPU Inference Performance)

Location: Hybrid in San Francisco or New York, United States

Salary: $240K–$270K plus equity

Company

Baseten provides inference infrastructure, applied AI research, and developer tooling that help AI companies bring advanced models into production.

What you will do

• Lead, mentor, hire, and develop a team of inference performance engineers.
• Own the technical roadmap and execution for inference runtime performance, balancing customer needs, model launches, and platform investments.
• Review designs and guide GPU profiling, optimization, benchmarking, and performance analysis.
• Drive productionization of quantization, speculative decoding, KV-cache reuse, chunked prefill, and custom scheduling.
• Bring up and tune new model architectures on new hardware and measure improvements in throughput, utilization, latency, and cost.
• Partner with Infrastructure, Inference Platform, Kernels, Model APIs, and customer-facing teams to coordinate launches and maintain operational excellence.

Requirements

• Bachelor’s, Master’s, or Ph.D. in Computer Science, Engineering, Mathematics, or a related field.
• Experience managing engineers, including hiring, mentoring, feedback, and performance reviews.
• Experience leading or closely supporting GPU optimization teams in training, inference, or recommendation systems.
• Strong technical understanding of GPU architecture, workloads, and performance tradeoffs.
• Familiarity with PyTorch, TensorRT, or TensorRT-LLM.
• Experience delivering complex technical roadmaps with strong written and verbal communication.

Nice to have

• Experience with vLLM, SGLang, or TensorRT-LLM.
• Production experience with LLM optimization techniques such as quantization, speculative decoding, or continuous batching.
• Experience with GPU kernels using CUDA, Triton, CUTLASS, or similar technologies.
• Experience scaling an engineering team through rapid startup growth.
• Background as a hands-on performance or systems engineer.

Culture & Benefits

• Competitive compensation with meaningful equity.
• U.S. employees and dependents receive full medical, dental, and vision insurance coverage.
• Flexible PTO and a company-wide winter break.
• Paid parental leave and a fertility and family-building stipend.
• U.S. employees have access to a company-facilitated 401(k).
• Exposure to ML startups and opportunities for learning and networking.