Senior Inference Runtime Engineer (AI)

TL;DR

Senior Inference Runtime Engineer (AI): Designing and optimizing the performance-critical serving layer of a MaaS platform with an accent on LLM runtimes, GPU utilization, and latency reduction. Focus on optimizing KV cache behavior, speculative decoding, and scaling distributed inference for high-throughput production environments.

Location: Singapore, Penang (Malaysia), or Taiwan

Company

A world-leading technology company specializing in Bitcoin mining and AI cloud infrastructure.

What you will do

• Optimize prefill/decode scheduling, continuous batching, and KV cache behavior for self-hosted LLMs.
• Tune vLLM, SGLang, and TensorRT-LLM runtimes to improve latency, throughput, and GPU utilization.
• Profile bottlenecks across GPU memory, HBM bandwidth, NCCL network, and model worker paths.
• Lead high-value model onboarding, including runtime selection, quantization, and parallelism strategies.
• Define runtime playbooks for reasoning, tool calling, and multimodal capabilities.
• Collaborate with SRE and performance engineers to translate benchmark findings into production improvements.

Requirements

• 6+ years of experience in systems, ML infrastructure, or high-performance backend engineering.
• Hands-on experience with LLM serving runtimes such as vLLM, TensorRT-LLM, SGLang, or Triton.
• Deep understanding of GPU memory, CUDA/NCCL, and distributed inference tradeoffs.
• Proficiency in Go or Python, with the ability to read runtime source code and profile traces.
• Experience operating production inference services with strict latency, availability, and cost targets.

Culture & Benefits

• Culture that values authenticity and diversity of thoughts and backgrounds.
• Inclusive environment with open workspaces and a startup spirit.
• Opportunity to network with industry pioneers and impact the future of the digital asset industry.
• Personal accountability, autonomy, and fast growth opportunities.
• Attractive welfare benefits, training, and mentoring.