Staff Software Engineer, Foundation Model Inference (AI Engineering)

TL;DR

Staff Software Engineer, Foundation Model Inference (AI Engineering): Building and scaling the unified serving layer for large language models (FMAPI) across real-time and batch inference with an accent on distributed AI workloads and GPU orchestration. Focus on improving reliability, latency, and efficiency of LLM infrastructure to power enterprise-scale AI applications.

Location: San Francisco, California

Salary: $190,000 — $265,000 USD

Company

Databricks is the data and AI company providing a Data Intelligence Platform to unify data, analytics, and AI for over 10,000 organizations worldwide.

What you will do

• Build LLM infrastructure for large-scale inference using partner models (OpenAI, Anthropic, Gemini) and self-hosted models (Qwen, GPT-OSS, Llama).
• Shape the FMAPI product roadmap and execution through direct engagement with enterprise users and model providers.
• Optimize the reliability, latency, and efficiency of distributed AI workloads.
• Collaborate with platform, infrastructure, and ML teams to deliver seamless end-to-end experiences.
• Design how developers and data scientists build and interact with AI on the Databricks platform.

Requirements

• 8+ years of experience in backend or infrastructure engineering.
• Expertise in distributed systems, scalable APIs, or cloud-native infrastructure.
• Strong programming skills in Scala, Go, or Python.
• Experience with real-time serving, ML infrastructure, or GPU orchestration.
• Strong product ownership mindset with a focus on shipping user-facing value.
• Must be based in San Francisco, California.

Nice to have

• Exposure to platforms like SageMaker, Vertex AI, or Azure ML.
• Experience building products that support AI workflows.

Culture & Benefits

• Opportunity to work in a high-agency, high-visibility team at the frontier of AI infrastructure.
• Inclusive culture committed to diversity and equal employment opportunity standards.
• Comprehensive benefits and perks tailored to the employee's region.
• Work on a platform used by over 50% of the Fortune 500.