Senior Big Data Engineer (Spark)

TL;DR

Senior Big Data Engineer (Spark/Kubernetes): Designing and optimizing scalable data processing pipelines for a data collaboration platform with an accent on high-performance infrastructure and real-time use cases. Focus on automating large-scale Kubernetes environments, ensuring system reliability, and driving the technical roadmap for AI/ML workloads.

Location: San Francisco, CA (Hybrid / options for working from home)

Salary: $130,000 – $196,500

Company

hirify.global is a data collaboration platform specializing in consumer privacy, data ethics, and foundational identity for innovative global companies.

What you will do

• Design, build, and optimize scalable batch and real-time data processing pipelines using Spark and Airflow.
• Lead the delivery of secure Big Data infrastructure using Dataproc, Redpanda, and Temporal.
• Automate large-scale Kubernetes infrastructure using Terraform and other infrastructure-as-code patterns.
• Define monitoring, alerting, and runbooks using Grafana to ensure operational excellence and high availability.
• Mentor vendor teams and external partners to extend and support Big Data solutions.
• Drive the technical roadmap by evaluating emerging technologies and building proofs of concept.

Requirements

• 7+ years of experience in software or data engineering with a focus on distributed systems.
• Hands-on expertise with Apache Spark and orchestration frameworks like Airflow or Temporal.
• Proficiency with Kubernetes and Terraform for provisioning production services.
• Experience with streaming and messaging technologies such as Redpanda or Kafka.
• Practical experience running Big Data workloads on cloud platforms (e.g., Dataproc).
• Must be based in or able to work from San Francisco, CA.

Nice to have

• 3+ years of software development in Python, Go, Java, or Scala.
• Experience using AI tools, NLP, and agent creation.
• Exposure to data platform security, encryption, and compliance standards.
• Experience mentoring engineers or leading technical initiatives.

Culture & Benefits

• Comprehensive medical, dental, and vision insurance.
• 401(k) matching plan (1:1 match up to 6% of salary).
• Flexible paid time off, paid holidays, and paid parental leave.
• Hybrid work flexibility to support work-life harmony.
• Collaborative environment with regular in-person and virtual social events.