Data Platform & Data Engineering (AI)

TL;DR

Senior/Lead/Principal Data Platform & Data Engineering (AI): Building and optimizing large-scale data pipelines, observability systems, and compute infrastructure for Spark workloads with an accent on distributed systems design, data quality, and scalable AI infrastructure. Focus on solving complex distributed systems challenges, improving monitoring and alerting, and exploring efficient ways to run smaller language models.

Location: Onsite in San Francisco, California

Company

hirify.global is a leading cloud-based software company providing customer relationship management services and enterprise applications.

What you will do

• Design and own large-scale data pipelines and observability systems that power metrics, logging, and real-time insights.
• Optimize core Spark performance and solve distributed systems challenges for big data compute infrastructure.
• Build scalable AI infrastructure, including exploring efficient ways to run smaller language models.
• Improve monitoring and alerting to ensure data quality and system visibility at scale.
• Operate mission-critical data platforms and infrastructure services at an enterprise scale.

Requirements

• Related technical degree required.
• Strong understanding of distributed systems design, including scalability, fault tolerance, and consistency.
• 7+ years of backend software development experience building large-scale distributed systems.
• Strong programming skills in Java (preferred); Python or Rust are strong pluses.
• Experience designing and operating large-scale data pipelines, ETL workflows, or streaming data systems.
• Experience with big data and data platform technologies such as Spark, Flink, Kafka, Trino, or HBase.
• Strong experience with public cloud platforms (AWS or GCP) and Kubernetes.
• Experience building or operating observability systems, telemetry pipelines, or monitoring platforms.

Culture & Benefits

• Wellbeing reimbursement.
• Generous parental leave, adoption assistance, and fertility benefits.
• Opportunities within a large, established engineering organization.
• Agile development practices.