Sr. Inference Service SRE & Automation Engineer (AI)

Location: Singapore, SG or Penang, Malaysia

Company

hirify.global is a technology company focused on Bitcoin mining solutions, AI cloud services, ASIC hardware, and high-performance computing datacenters.

What you will do

• Own end-to-end reliability for customer inference endpoints, including API availability, request success rate, latency, and the published SLA.
• Operate the inference serving path across API gateways, routers, schedulers, queues, prefill/decode services, KV cache, inference engines, and GPU model replicas.
• Manage Kubernetes, multi-cluster serving across Malaysia, Singapore, and Japan, cross-cluster traffic steering, failover, and regional recovery.
• Operate GPU serving fleets and automate monitoring, node draining, cordoning, replica replacement, and fault isolation across GPU, NVLink, and NCCL failures.
• Define SLI/SLO/SLA systems, error budgets, burn-rate alerting, synthetic probes, incident processes, customer communication, and postmortems.
• Build change-safety and resilience practices, including canary releases, automatic rollback, failover drills, disaster recovery tests, fault injection, and backup restore verification.

Requirements

• 5+ years of SRE or production platform operations experience, including 2+ years operating customer-facing API services against a published SLA.
• Deep Kubernetes operations experience covering HA control planes, etcd, CNI, service discovery, ingress, edge tiers, and multi-cluster topologies.
• Production experience with LLM or model inference using vLLM, SGLang, TensorRT-LLM, Triton, or equivalent runtimes.
• Experience operating GPU fleets and diagnosing DCGM, Xid, ECC, NVLink, and NCCL failures, including node drain and replica recovery.
• Strong SRE fundamentals covering SLI/SLO/SLA design, incident command, postmortems, capacity planning, overload protection, and stateful dependency failover.
• Proficiency in Terraform, Helm, GitOps workflows such as ArgoCD or Flux, and Go or Python for automation, controllers, and operators.

Nice to have

• AIOps experience and a runbook-as-code approach to automated remediation.
• Experience with VictoriaMetrics, VictoriaLogs, Tempo, Pyroscope, Grafana, ClickHouse, or comparable observability systems.
• Experience with customer-facing incident communication and SLA reporting.

Culture & Benefits

• Inclusive environment that values authenticity and diverse perspectives.
• Startup-oriented environment within a fast-growing technology company.
• Autonomy, personal accountability, learning opportunities, training, and mentoring.
• Opportunity to contribute to digital asset, AI cloud, and high-performance computing initiatives.
• Welfare benefits and opportunities for professional development.