Software Engineer (Observability)

TL;DR

Software Engineer (Observability): Building and maintaining observability infrastructure for monitoring and optimizing system performance with an accent on scalability, reliability, and performance. Focus on designing telemetry pipelines, developing APIs, and integrating observability into internal platforms.

Location: Hybrid (Palo Alto, CA and San Francisco, CA)

Company

xAI creates AI systems that understand the universe and aid humanity in its pursuit of knowledge.

What you will do

• Design and implement scalable observability infrastructure for metrics, logging, and tracing.
• Build high-performance telemetry pipelines that handle massive ingestion volumes.
• Develop APIs, query engines, and UIs for real-time insights.
• Define and enforce best practices for instrumentation and alerting.
• Partner with infrastructure and product teams for integration.
• Own the reliability and performance of the observability stack.

Requirements

• Production-level proficiency in Go, Rust, Scala, or similar languages.
• Deep understanding of distributed systems and telemetry architecture.
• Experience building and operating infrastructure at scale.
• Familiarity with observability stacks like Prometheus and Grafana.
• Experience with Kafka, Redis, or large-scale time series databases.
• Experience operating observability pipelines in Kubernetes.

Nice to have

• Experience with ClickHouse or VictoriaMetrics.

Culture & Benefits

• Equity and comprehensive medical, vision, and dental coverage.
• Access to a 401(k) retirement plan.
• Short & long-term disability insurance and life insurance.
• Various discounts and perks.

Hiring process

• 15-minute phone interview followed by technical interviews.
• Practical coding assessment and systems design hands-on session.
• Project deep-dive presentation and meet with the wider team.