Senior Software Engineer - Data Pipelines (Python/AWS)

TL;DR

Senior Software Engineer - Data Pipelines (Python/AWS): Building and operating distributed data pipelines across 15+ regions with an accent on large-scale processing, data quality, and observability. Focus on optimizing PySpark workloads, evolving Snowflake data models, and designing reliable production systems for threat intelligence, asset, and detection data.

Location: Hybrid work from the Tel Aviv office, Israel

Company

A global cybersecurity company providing exposure management solutions that help organizations understand and reduce cyber risk.

What you will do

• Own the full lifecycle of production data pipelines across 15+ regions, from design through operations.
• Set technical direction for pipeline architecture and write high-performance distributed processing jobs.
• Build and maintain pipelines supplying threat intelligence, asset, and detection data to product access layers.
• Evolve underlying data models and design frameworks for data health, conditional validation, and quality monitoring.
• Lead observability across pipelines through distributed tracing, structured logging, and metrics.
• Raise engineering standards through design reviews, code reviews, mentorship, and automation of on-call operations.

Requirements

• 7+ years of experience building and operating production data pipelines at scale.
• Advanced Python experience, including Pydantic, asynchronous programming, decorators, packaging, and profiling.
• Production Apache Airflow expertise, including large-scale DAG authoring, dynamic task mapping, deferrable operators, SLAs, and platform operations.
• Strong PySpark optimization skills and experience with EMR or Databricks, plus deep Snowflake knowledge.
• Experience with AWS services including S3, SQS/SNS, EMR, IAM, and infrastructure as code.
• Hands-on experience with data quality frameworks, observability, CI/CD, staged deployments, pytest, and integration testing.

Culture & Benefits

• End-to-end ownership with minimal bottlenecks and strong engineering autonomy.
• AI-native development supported by custom plugins and protocols.
• High engineering rigor with dead-code detection, continuous security scanning, and production monitoring.
• On-call responsibilities supported by automation tools and an expectation of continuous improvement.
• Collaborative culture focused on belonging, respect, excellence, and shared results.