Principal Engineer, AI Cloud Software

Location: Singapore

Company

hirify.global Technologies develops and operates energy-efficient AI infrastructure across Asia Pacific, including the hirify.global AI Cloud GPU platform for training and deploying AI models.

What you will do

• Define GPU and host health criteria and service-readiness gates for customer and internal workloads.
• Build and publish reusable dashboards, alerts, PromQL and LogQL queries, health checks, and diagnostic guidance.
• Develop pass/fail validation using DCGM, NCCL and bandwidth tests, stress testing, burn-in, and validation jobs.
• Diagnose and isolate GPU, host, network, cooling, power, and management-interface failures across large GPU systems.
• Detect non-crashing failures such as ECC errors, NVLink retries, thermal throttling, XID patterns, silent data corruption, and stragglers.
• Partner with commissioning, operations, infrastructure, telemetry, and customer teams; participate in incidents and convert findings into checks, alerts, and runbooks.

Requirements

• Bachelor’s degree in computer science or a related technical field, or equivalent practical experience.
• 7+ years of experience in GPU, HPC, AI infrastructure, or large-scale systems engineering, including ownership of health monitoring or diagnostics.
• Production experience diagnosing GPU faults involving XID events, ECC and memory errors, NVLink, power, or thermal limits.
• Strong Linux and server fundamentals, with the ability to debug across GPU, CPU, memory, PCIe, power, and cooling systems.
• Hands-on experience with DCGM, NCCL or collective tests, stress testing, infrastructure telemetry, PromQL, LogQL, Grafana, or comparable tools; automation skills in Python or similar.
• Clear written and verbal communication in English; willingness to participate in incident-response on-call and occasional overseas travel.

Nice to have

• Experience with large production GPU systems, tight GPU-to-GPU interconnects, multiple sites, or multiple GPU generations.
• Background in an AI cloud, GPU cloud, or HPC center running customer workloads.
• Experience with GPU or host failure prediction and structured operational knowledge for AI-assisted incident recovery.

Culture & Benefits

• Work on sustainable, energy-efficient AI infrastructure and large-scale GPU cloud systems.
• Collaborate across commissioning, operations, infrastructure, telemetry, engineering, and customer teams.
• Contribute to a diverse and inclusive workplace.