Distributed LLM Inference Engineer (AI)

Location: Hybrid in San Francisco or Palo Alto, United States

Salary: $170,000–$245,000 annual target base salary, plus equity

Company

hirify.global commercializes Ray, an open-source distributed computing platform for scalable machine learning applications.

What you will do

• Develop and ship end-to-end batch and online inference solutions for open-source Ray users and hirify.global customers.
• Integrate Ray Data with LLM engines and implement optimizations for large-scale machine learning inference.
• Integrate open-source technologies such as vLLM into hirify.global solutions and contribute improvements upstream.
• Implement and extend best practices from the open-source and research communities.

Requirements

• Experience running machine learning inference at large scale with high throughput and low latency.
• Familiarity with deep learning and frameworks such as PyTorch.
• Solid understanding of distributed systems and machine learning inference challenges.
• Ability to work in a hybrid arrangement in San Francisco or Palo Alto.

Nice to have

• Machine learning systems knowledge and experience using Ray.
• Experience with LLM engines such as vLLM or TensorRT-LLM.
• Contributions to PyTorch, TensorFlow, Triton, TVM, or MLIR.
• Prior experience working with GPUs or CUDA.

Culture & Benefits

• Stock options and equity participation.
• Healthcare premiums covered at 99% for employees and dependents.
• 401(k) retirement plan, education and wellbeing stipend, and paid parental leave.
• Fertility benefits, paid time off, commute reimbursement, and covered in-office meals.