Senior Software Engineer (Spark Platform)

Location: Hybrid; must be located in San Francisco, Sunnyvale, Seattle, or New York City, or be willing to relocate to the Bay Area, Seattle, or NYC.

Salary: $130,600–$192,000, $159,800–$235,000, or $193,800–$285,000 USD annually, depending on work location and level.

Company

hirify.global is a technology and delivery platform that operates large-scale data, analytics, and machine learning systems.

What you will do

• Set the multi-year technical direction for the in-house Spark-on-Kubernetes platform, including runtime, shuffle, scheduling, and reliability.
• Own distributed-systems challenges involving shuffle architecture, multi-tenant scheduling, runtime performance, and large-scale failure modes.
• Partner with the Engineering Manager on technical roadmaps, hiring, interview design, and team structure.
• Lead design reviews, mentor engineers, and raise engineering quality across the platform team.
• Represent Spark Platform in architecture forums and collaborate with Data Engineering, ML Platform, and product engineering teams.

Requirements

• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep hands-on experience with Apache Spark internals, query execution, shuffle, and the executor/driver model in a platform operations context.
• Production experience operating Kubernetes workloads, including operator patterns, executor pod lifecycle, network topology, and multi-tenant failure modes.
• Experience with remote shuffle systems, big-data schedulers, or observability and SRE practices for distributed compute platforms.
• Professional experience with Scala, Java, Python, or Go, plus strong SQL skills.
• B.S., M.S., or PhD in Computer Science or equivalent; demonstrated technical leadership, mentoring, and roadmap ownership.

Nice to have

• Experience with Amazon EMR, Databricks, Celeborn, Magnet, Cosco, YuniKorn, Volcano, Kueue, or the Spark-on-Kubernetes operator.
• Familiarity with Apache Iceberg, Delta Lake, and query or SQL engines that read these formats.

Culture & Benefits

• Hybrid work environment with location-based compensation.
• Equity grant opportunities in addition to base salary.
• 401(k) with employer matching and comprehensive medical, dental, and vision coverage.
• Flexible paid time off, paid sick leave, 11 paid holidays, and 16 weeks of paid parental leave.
• Wellness, commuter, disability, life insurance, family-forming, and mental health benefits.