Staff Software Engineer (Big Data)

TL;DR

Staff Software Engineer (Big Data): Designing and optimizing complex streaming and batch data pipelines with an accent on scalable cloud software architecture and event-driven orchestration. Focus on migrating cron-based pipelines to modern frameworks and implementing Lakehouse architectures using Delta Lake and Apache Iceberg.

Location: Singapore

Company

hirify.global delivers AI-powered automation and network assurance solutions for enterprise communications technology.

What you will do

• Design, build, and optimize complex streaming and batch pipelines using Scala, Java, Python, and Apache Spark.
• Lead the migration of cron-based Spark pipelines to event-driven orchestration frameworks like Argo Workflows, Apache Airflow, or Temporal.
• Manage high-frequency data ingestion and schema evolution using Apache Kafka and Google Protocol Buffers (GPB).
• Implement and optimize data storage using modern open table formats like Delta Lake and Apache Iceberg.
• Deploy, scale, and tune workloads on Google Kubernetes Engine (GKE) using Helm and the Spark K8s Operator.

Requirements

• 5+ years of software engineering experience focusing on distributed systems, big data, or backend engineering.
• Proficiency in Scala or other JVM languages.
• Deep experience with Apache Spark, including tuning, partitioning, and memory management.
• Solid understanding of Apache Kafka (topics, partitions, consumer groups) and Protobuf.
• Hands-on experience deploying and managing applications on Kubernetes (ideally GKE) using Helm.
• Experience with modern data orchestration tools such as Airflow, Argo, Dagster, Prefect, or Temporal.

Nice to have

• Experience writing high-performance code in Rust.
• Production experience with Delta Lake or Apache Iceberg metadata and partition pruning.
• Familiarity with emerging streaming engines like Arroyo or Apache Flink.
• Experience building distributed load-testing frameworks or chaos engineering tools for data pipelines.

Culture & Benefits

• Opportunity to work at the intersection of AI and Networking.
• Collaborative environment with a global team of innovators.
• Equal Opportunity Employer focusing on diversity and inclusion.