Principal Engineer, Observability (AI Infrastructure)

Location: Singapore; hybrid schedule with 4 days in the office and 1 day working from home

Company

hirify.global Technologies develops and operates sustainable AI infrastructure and hirify.global AI Cloud, a large-scale GPU cloud platform across Asia Pacific.

What you will do

• Own the architecture and technical direction of the observability platform across data centres and cloud regions.
• Design telemetry collection, streaming, storage, querying, alerting, and application layers that scale to millions of samples per second and hundreds of thousands of GPUs.
• Define telemetry standards and enable teams to create dashboards, alerts, and SLOs independently through templates and infrastructure-as-code.
• Build the live topology model covering sites, racks, GPUs, networks, cooling systems, and workloads.
• Deliver reliable telemetry to internal teams, customers, and partner platforms with versioned interfaces, tenant isolation, and auditability.
• Lead observability SLOs, incident response, fault injection, post-mortems, architecture reviews, mentorship, and technical interviews.

Requirements

• Bachelor’s degree in computer science or a related technical field, or equivalent practical experience.
• 10+ years in software, infrastructure, or platform engineering, including at least 3 years at principal level or equivalent.
• Experience designing and operating metrics, logs, and traces platforms at millions of samples per second using Prometheus-compatible stores such as VictoriaMetrics, Thanos, or Mimir.
• Experience with streaming telemetry ingestion using Kafka or Redpanda, Vector or OpenTelemetry Collector, and production Kubernetes for high-throughput stateful services.
• Strong Go or Python skills, plus experience with OpenTelemetry or OpenMetrics, hardware or network telemetry, and systems such as Redfish, gNMI/OpenConfig, or DCGM.
• Production security experience under SOC 2 Type 2, ISO 27001, or equivalent controls; clear written and verbal English communication and willingness to join on-call rotations.

Nice to have

• Experience with AI cloud or high-performance computing providers, GPU health checks, or fleet failure prediction.
• Facilities and operational technology telemetry, including OPC UA, Modbus, SCADA, or building management systems.
• Digital twins, live topology models, customer-facing telemetry, tenant isolation, or telemetry-driven incident analysis.
• Kubernetes operators, relevant certifications, or experience leading secure AI-assisted engineering.

Culture & Benefits

• Hybrid work arrangement with 4 office days and 1 work-from-home day in Singapore.
• Work on sustainable AI infrastructure, liquid cooling, energy management, and large-scale GPU cloud systems.
• Occasional overseas travel is required.
• Inclusive workplace supporting candidates from diverse backgrounds.

Hiring process

• Participate in technical interviews and assessments for engineering candidates as part of the role.