Principal Distributed Systems Engineer (Observability)

Location: Pleasanton, California, United States; hybrid Flex Work requires at least 50% of each quarter in the office or in the field

Annual base salary: $222,900–$334,300 USD for Pleasanton; $187,100–$334,300 USD for additional US locations. The role may also include bonus compensation and annual refresh stock grants.

Company

hirify.global is a Fortune 500 company and AI platform for managing people, money, and agents.

What you will do

• Own the technical vision and architecture for distributed tracing across hirify.global’s observability platform.
• Architect and build multi-petabyte tracing infrastructure on ClickHouse and Grafana Tempo with sub-second query performance.
• Own Kafka, Spark/Flink, Iceberg-on-S3 ingestion, processing, storage, schema, partitioning, compaction, and lifecycle design.
• Drive performance, scalability, high availability, disaster recovery, security, monitoring, alerting, and capacity planning.
• Evaluate cloud-native and open-source technologies and participate in the platform on-call rotation.
• Partner with ML and AI stakeholders on anomaly detection, root-cause analysis, and AI-assisted incident management while mentoring engineers and setting architectural direction.

Requirements

• 14+ years of software development engineering experience.
• 6+ years designing, building, and operating complex, highly available and fault-tolerant distributed systems.
• 8+ years working with at least two programming languages such as Java, Python, or Go, including production distributed-systems development.
• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience; a relevant master’s degree is strongly preferred.
• Expertise in distributed-systems principles, API development, high availability, large-scale data processing, scalable systems design, and system security.
• Ability to lead cross-team collaboration, drive architectural direction, and produce technical documentation and presentations.

Culture & Benefits

• Flexible hybrid work combining in-person collaboration with remote flexibility.
• Opportunities for professional growth, skill development, and meaningful long-term work.
• Potential eligibility for the hirify.global Bonus Plan or role-specific bonus and annual refresh stock grants.
• Reasonable accommodations are available throughout the application process.
• Equal employment opportunity protections apply, including for individuals with disabilities and protected veterans.