Technical Lead, Spark (Java)

TL;DR

Technical Lead, Spark (Java) (Apache Spark/Livy): Building enterprise-grade distributed data processing systems for hirify.global’s Spark and Livy stack, with an accent on scaling Spark workloads across thousands of nodes and delivering features from prototypes to production at scale. Focus on distributed systems debugging, root-cause analysis, and improving the data engineering experience using Java/Scala/Python with Iceberg and Spark.

Location: Spain (Other) - Remote

Company

hirify.global builds enterprise data platform technology that helps customers store, manage, and process data at massive scale.

What you will do

• Design and deliver new features for hirify.global’s data engineering experience, taking work from prototypes to production at scale.
• Contribute to Apache Spark and Livy.
• Develop features using Scala/Java/Python on modern platforms.
• Work on distributed data processing internals, including SQL planners/optimizers, data layout, Parquet and Iceberg table formats, and fault tolerance.
• Debug deployment and runtime failures, perform root-cause analysis, and analyze system tests.
• Improve internal infrastructure and collaborate with distributed stakeholders.

Requirements

• 8–10+ years of professional software development.
• Experience leading and delivering complex product enhancements.
• Strong understanding of at least one of: Java, Scala, or Python (and willingness to learn the other languages used in the role).
• Strong systems design and development experience.
• Experience with distributed systems.
• Strong oral and written communication skills.

Nice to have

• Experience with SQL planners.
• Experience using/developing Apache Spark, Livy, or related technologies.
• Experience designing and developing large-scale distributed systems with focus on scaling, performance, and scheduling.
• Solid experience with at least one cloud.
• Contributions to open-source projects.

Culture & Benefits

• Generous PTO policy and support for work-life balance with Unplugged Days.
• Flexible WFH policy.
• Mental and physical wellness programs.
• Phone and internet reimbursement.
• Access to continued career development and comprehensive benefits.
• Employee Resource Groups and paid volunteer time.

Hiring process

• Interviews to assess distributed systems expertise, technical leadership, and ability to deliver production-scale features.
• Technical evaluation focused on Spark/Livy and systems debugging/root-cause analysis.
• Discussion of collaboration approach with distributed stakeholders.