Software Engineer, Model API Infra

Location: Mountain View or Redmond, United States

Base pay: USD $119,800–$234,700 per year across the U.S. for IC4; USD $142,800–$274,800 per year across the U.S. for IC5. San Francisco Bay Area and New York City metropolitan area ranges differ.

Company

Microsoft AI develops MAI models and AI platforms that deliver model capabilities to production products and services.

What you will do

• Design, implement, and operate scalable API services for text, multimodal, and emerging MAI models.
• Build infrastructure that maintains consistent model, tool, and harness behavior across training, evaluation, and production serving.
• Automate model deployment and validation using functional, safety, quality, and performance evaluations.
• Build autoscaling systems for model pools using latency SLOs, utilization, model characteristics, and capacity constraints.
• Develop telemetry, observability, and debugging tools for cross-system investigations and production reliability.
• Collaborate with Foundry, Copilot, inference, research, evaluation, and product teams to deliver model capabilities.

Requirements

• Bachelor’s degree in Computer Science or a related technical field and 5+ years of technical engineering experience, or equivalent experience.
• Strong coding experience with Python, C++, or similar languages.
• Experience with LLM inference and serving systems such as vLLM, SGLang, TensorRT-LLM, or custom inference engines.
• Experience with distributed serving concepts including TP, PP, DP, KV cache, and speculative decoding.
• Strong systems and debugging skills, including performance analysis on GPU clusters.
• Experience with deployment automation, Kubernetes orchestration, and cloud platforms.

Nice to have

• Master’s degree and 5+ years of relevant experience or equivalent experience.
• Experience deploying models to Azure ML, Foundry, Kubernetes, or containerized production platforms.
• Experience with multi-adapter serving, disaggregated prefill/decode, or large-scale KV cache management.
• Experience with Grafana, Prometheus, Kusto, or equivalent observability tools.
• Experience building LLM evaluation and benchmarking harnesses, or knowledge of GPU topology, memory management, and modern accelerator hardware.

Culture & Benefits

• Cross-functional collaboration with platform, research, evaluation, inference, and product teams.
• Potential eligibility for benefits and additional compensation.
• Equal opportunity employment and reasonable accommodation support during the application process.

Hiring process

• Applications are accepted on an ongoing basis until the position is filled.