Software Engineer (AI Inference)

Location: Hybrid in San Francisco, Montreal, New York, Seattle, or Toronto

Salary: $165,000–$330,000 per year, plus equity

Company

Baseten provides AI inference infrastructure, applied AI research, and developer tooling for companies bringing machine-learning models into production.

What you will do

• Evaluate and automate LLM quality benchmarks and custom performance suites for long-context, KV-cache reuse, and disaggregated serving workloads.
• Develop GPU-enabled internal development environments for high-performance model experimentation.
• Build and contribute to open-source benchmarking and model-evaluation tools, including InferenceMAX and genai-bench.
• Profile systems with PyTorch Profiler, NVIDIA Nsight Systems, and py-spy to identify compute and networking bottlenecks.
• Create real-time dashboards, alerts, CI/CD performance tests, and release automation for model runtimes.
• Develop optimization tools to identify the best latency, cost, and quality configuration for each model and workload.

Requirements

• Mid-to-senior engineering experience with strong technical depth and communication skills.
• Understanding of GPU memory subsystems, InfiniBand, and data movement across clusters.
• Experience or strong interest in scripting, stress testing, and fuzz testing to identify system limits.
• Curiosity about Transformer mathematics, FLOPs, and memory requirements.
• Familiarity with Python and willingness to master the NVIDIA software stack.
• Ability to drive cross-team initiatives, work through ambiguous requirements, and mentor engineers.

Nice to have

• Familiarity with C++.

Culture & Benefits

• High ownership and autonomy while leading a small team and building tools from scratch.
• Opportunities to contribute to open-source projects and develop expertise in GPU orchestration and LLM inference.
• Meaningful equity and competitive compensation.
• Flexible PTO, including a company-wide winter break, and paid parental leave.
• U.S. employees receive full medical, dental, and vision coverage, plus access to a company-facilitated 401(k).
• Fertility and family-building support through Carrot.