Principal Observability Platform Engineer (AI)

Location: US

Salary: $190,000–$300,000 USD base salary per year, with potential bonus, equity, and/or commission eligibility.

Company

Nscale provides high-performance GPU cloud infrastructure for AI startups and enterprise customers.

What you will do

• Own the technical strategy and architecture for observability across metrics, logs, traces, and alerting at scale.
• Define tooling, data models, ingestion patterns, retention policies, and cardinality-management strategies.
• Design platforms that identify systemic gaps, make failures visible, and accelerate diagnosis.
• Partner with SRE, infrastructure, and AI/ML teams to embed observability into engineering and operations.
• Establish engineering standards, evaluate observability tooling, and lead durable improvements from incident postmortems.
• Mentor and technically grow the observability team while raising engineering standards across the organization.

Requirements

• 8+ years of experience in SRE, infrastructure engineering, platform engineering, or observability-focused roles.
• Experience operating observability infrastructure at significant scale and designing for 10x growth.
• Deep hands-on experience with a significant subset of Prometheus, Thanos, VictoriaMetrics, Grafana, Loki, Tempo, OpenTelemetry, ClickHouse, or Elastic.
• Strong engineering fundamentals and proficiency in Python, Go, or a similar language, with end-to-end ownership of complex systems.
• Experience with Kubernetes at scale and Infrastructure-as-Code using Terraform, Ansible, or an equivalent tool.
• Ability to architect systems, write and review code, explain trade-offs, and influence teams without direct authority.

Nice to have

• Experience with high-volume observability streaming pipelines using Kafka, Vector, Fluent Bit, or similar tools.
• Background in AI/ML infrastructure observability, including GPU utilization, training-job visibility, and inference latency.
• Familiarity with GPU infrastructure or HPC environments such as Slurm.
• Experience defining observability strategy at the organizational level.

Culture & Benefits

• Culture centered on innovation, ownership, accountability, openness, and transparency.
• Medical, dental, and vision benefits.
• Flexible paid time off and parental leave.
• Retirement plan participation.
• Potential bonus, equity, and/or commission programs in addition to base salary.