Software Engineer (Spark Platform)

Location: Hybrid; must be located in San Francisco, Sunnyvale, Seattle, or New York City, or be willing to relocate to the Bay Area, Seattle, or NYC.

Salary: $130,600–$192,000, $159,800–$235,000, or $193,800–$285,000 USD annually, depending on level and location, plus equity opportunities.

Company

hirify.global operates a large-scale technology platform for delivery and commerce services.

What you will do

• Build and operate an in-house Apache Spark platform covering runtime, scheduling, reliability, and user-facing tooling.
• Design multi-tenant scheduling, executor bin-packing, and cost-aware workload placement for multiple consumer teams.
• Automate cluster provisioning, upgrades, capacity changes, and node-failure handling.
• Develop observability and incident automation for end-to-end platform debugging and sustainable on-call operations.
• Partner with senior engineers and platform consumers on shuffle, runtime, architecture, and reliability initiatives.

Requirements

• B.S., M.S., PhD in Computer Science, or equivalent experience.
• Experience operating production distributed systems and Apache Spark at scale through Amazon EMR, Databricks, or an in-house deployment.
• Hands-on experience with production Kubernetes, including controllers, operators, custom resources, and multi-tenant cluster failure modes.
• Knowledge of batch or big-data schedulers and/or the Spark-on-Kubernetes operator.
• Experience with observability tools such as Prometheus, OpenTelemetry, distributed tracing, structured logging, and SLOs/SLIs.
• Professional experience with Python, Go, Scala, or Java; SQL fluency; and familiarity with AWS infrastructure.

Nice to have

• Experience with YuniKorn, Volcano, Kueue, or equivalent schedulers.
• Experience with VPC networking, instance lifecycle management, spot or preemptible markets, and autoscaling.

Culture & Benefits

• Hybrid work environment with collaboration across platform teams and company-wide consumers.
• 401(k) plan with employer matching, medical, dental, and vision insurance.
• Sixteen weeks of paid parental leave, family-forming assistance, and mental health support.
• Flexible paid time off, paid sick leave, and eleven paid holidays.
• Wellness, commuter benefits, disability, and basic life insurance.