Software Engineer, Observability (AI)

Location: Memphis, Tennessee, United States; in-person collaboration with users is required.

Company

xAI is building a full super-intelligence stack spanning large-scale compute infrastructure, Grok models, and AI applications.

What you will do

• Design and implement scalable, fault-tolerant real-time data ingestion pipelines for telemetry from compute nodes, GPUs, accelerators, schedulers, storage, networks, and facility systems.
• Develop frontend visualizations, interactive dashboards, and UI components for operators, capacity planners, and engineering stakeholders.
• Build and operate production systems covering UIs, APIs, data pipelines, authentication, uptime monitoring, predictive alerting, risk management, and site-wide impact assessments.
• Maintain repair trackers, vendor RMA workflows, operational infrastructure, and integrations with ticketing, inventory, telemetry, and vendor systems.
• Automate cluster monitoring and improve anomaly detection while maintaining uptime, data integrity, reconciliation, access control, and safe deployments.
• Work in person with users, investigate operational problems, and support GPU uptime improvements; participate in a weeklong rotational on-call schedule and work extended hours or weekends when required for mission-critical deadlines.

Requirements

• Bachelor's degree in computer science, engineering, mathematics, or another STEM discipline plus 3+ years of professional software development experience, or 5+ years of professional software development experience in lieu of a degree.
• Programming experience in Python, Rust, C++, Java, or similar languages.
• Experience with Postgres, MySQL, MongoDB, DynamoDB, or similar database management systems.
• Proven ownership of correctness-sensitive systems such as state machines, workflows, or operational data where inaccurate state has real-world impact.
• Experience collaborating with cross-functional teams.
• Ability to work extended hours or weekends as needed for mission-critical deadlines.
• Ability to participate in a weeklong rotational on-call schedule.

Nice to have

• Experience building or operating large-scale observability, telemetry, or real-time analytics systems.
• Experience with time-series or analytical databases such as Prometheus, VictoriaMetrics, ClickHouse, or InfluxDB, and streaming or high-volume telemetry pipelines.
• Experience building observability dashboards and alerting with Grafana, New Relic, Splunk, or similar tools.
• Frontend experience with React, Angular, or a comparable JavaScript framework for critical-operations interfaces.
• Experience designing and shipping multi-service mission-critical systems with RESTful APIs, data stores, and at least one of UI, pipelines, or authentication.
• Production experience operating services on cloud or on-premises infrastructure with Docker, Kubernetes, and automated CI/CD deployments such as ArgoCD or GitHub Actions.
• On-call discipline and end-to-end ownership of internal platforms in rapidly changing environments.
• Experience with HPC or supercomputing environments, including Slurm, GPU or accelerator telemetry, high-speed fabrics, or parallel filesystems.
• Experience integrating industrial or OT data sources, including Ignition, tag historians, Modbus, OPC-UA, or power and cooling infrastructure telemetry.
• Master's degree in computer science or a related field.

Culture & Benefits

• Work on software that supports frontier AI training and inference at large, heterogeneous, continuously operating datacenters.
• Collaborate directly with datacenter operations, AI research, and facility infrastructure teams.
• Work closely with users in person to incorporate feedback and improve operational processes.
• Operate systems with production rigor, emphasizing reliability, auditability, automation, and rapid incident response.