Staff Software Engineer (Distributed Systems)

TL;DR

Staff Software Engineer (Distributed Systems): Building and optimizing the persistence layer for Temporal Visibility, a system powering workflow listing, filtering, and observability across massive state machines, with an accent on high write throughput, complex secondary indexing, and operational continuity. Focus on re-architecting for dramatic performance and scalability, leading large-scale data migrations, and ensuring system reliability.

Location: Remote in the United States

Salary: $185,000 - $270,000 USD

Company

Temporal is an open-source programming model company focused on simplifying code and making applications more reliable for developers.

What you will do

• Lead the re-architecture and implementation of Temporal Visibility's persistence layer for dramatically improved performance, scalability, and operability.
• Design and execute safe, large-scale online data migrations, including strategies for dual-write, backfill, and cutover, without customer downtime.
• Define, own, and improve SLOs, profile hot paths, and lead systematic performance tuning efforts for Visibility storage and query paths.
• Build operational playbooks, dashboards, and alerting to ensure the system is understandable and debuggable for on-call engineers.
• Provide technical leadership, break down large initiatives into executable phases, and mentor other engineers in persistence and storage domains.
• Partner closely with Server, Cloud, and Developer Experience teams to deliver end-to-end features.

Requirements

• 5+ years of experience as an arranger and/or builder/enhancer of highly scalable distributed systems.
• Strong computer science fundamentals in distributed systems, including concurrency, consistency models, and failure modes.
• Significant experience writing and operating concurrent production systems in Go, Java, or similar languages.
• Hands-on experience designing, operating, and tuning ClickHouse and/or Elasticsearch, ideally in self-hosted environments.
• Experience building and running services on AWS.
• Demonstrated ability to lead large, multi-quarter technical initiatives, especially involving core data infrastructure and live data migrations.

Nice to have

• Prior contributions to Temporal, Cadence, or other workflow engines.
• Deep expertise in storage internals (e.g., columnar stores, LSM trees, inverted indexes, transactional logs).
• Experience operating multi-region services with ≥99.99% uptime.
• Strong background in operating and evolving Open Source systems.
• Experience building Kubernetes controllers and/or CRDs.

Culture & Benefits

• Unlimited PTO, 12 Holidays + 2 Floating Holidays.
• 100% Premiums Coverage for Medical, Dental, and Vision (for US employees).
• Empower 401K Plan (for US employees).
• Additional perks including Learning & Development, Lifestyle Spending Account ($1,200/year), In-Home Office Setup ($1,000/year), WFH Meals ($3,600/year), and Internet Stipend ($74/month).
• Globally distributed team that values opportunities for occasional in-person connection and travel for company events.
• Calm App Subscription for Mental Health & Wellness.