Senior Software Engineer, Spark Platform

Location: Hybrid; must be located in San Francisco, Sunnyvale, Seattle, or New York City, US

Salary: $159,800–$235,000 USD base salary, plus potential equity grants

Company

hirify.global operates a large-scale Apache Spark ecosystem supporting data, analytics, and machine learning workloads.

What you will do

• Set the multi-year technical direction for the in-house Spark-on-Kubernetes platform, including runtime, shuffle, scheduling, and reliability.
• Own distributed-systems challenges involving shuffle architecture, multi-tenant scheduling, runtime performance, and large-scale failure modes.
• Partner with the Engineering Manager on technical roadmaps, hiring, interview design, and team structure.
• Provide technical leadership through design reviews, mentorship, and engineering quality improvements.
• Represent Spark Platform in cross-team architecture discussions with Data Engineering, ML Platform, and product engineering.

Requirements

• Bachelor’s, master’s, or PhD in Computer Science or equivalent experience.
• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep hands-on Apache Spark experience, including internals, query execution, shuffle, and the executor/driver model, with platform-scale operations experience.
• Production experience with Kubernetes, including operator patterns, executor pod lifecycle, network topology, and multi-tenant failure modes.
• Experience with remote or external shuffle systems, batch or big-data schedulers, or observability and SRE patterns for distributed compute platforms.
• Professional experience with Scala, Java, Python, or Go, plus strong SQL skills; familiarity with Apache Iceberg, Delta Lake, and query or SQL engines is expected.

Nice to have

• Experience with Apache Iceberg or Delta Lake and the query engines that read them.
• Experience operating Spark on Amazon EMR, Databricks, or an in-house deployment.

Culture & Benefits

• Hybrid work in one of the listed US locations.
• 401(k) plan with employer matching.
• Medical, dental, vision, disability, and basic life insurance.
• Paid parental leave, paid time off, paid sick leave, and 11 paid holidays.
• Wellness, commuter, family-forming assistance, and mental health benefits.