Lead Data Engineer (Ab Initio)

TL;DR

Lead Data Engineer (Ab Initio/Spark): Design, develop, and maintain end-to-end data pipelines for TV, Broadband, Digital Voice services using AWS, GCP, Cloudera with an accent on scalability, performance, and multi-cloud environments. Focus on building batch/streaming workloads, optimizing ETL/ELT jobs, and orchestrating data ingestion with Spark, NiFi for high-throughput processing.

Location: Hybrid - 3 days a week in office (London or Belfast, UK), 2 days from anywhere

Company

hirify.global, the world’s first telco and home to BT, EE, Openreach, Plusnet, is transforming UK digital infrastructure with full fibre broadband and 5G.

What you will do

• Design, develop, and maintain end-to-end data pipelines using Ab Initio (GDE, PDL, Conduct>IT, Express>IT).
• Build scalable batch and streaming workloads with Apache Spark (PySpark/Scala).
• Develop and orchestrate data ingestion flows using Apache NiFi for real-time use cases.
• Optimize ETL/ELT jobs, perform bottleneck analysis and tuning for Spark, NiFi, Ab Initio.
• Improve automated testing, CI/CD pipelines, and conduct demos/knowledge transfers to stakeholders.
• Line manage a small team of junior data engineers and support continual learning.

Requirements

• Proficient in Big Data solutions using Hadoop, GCP, or AWS.
• Strong expertise in Ab Initio (GDE, PDL, Conduct>IT, Express>IT, EME) with parallel processing and optimization.
• Hands-on with Apache Spark (SQL, Streaming, tuning) and Apache NiFi (flow design, tuning, security).
• Skilled in Scala, Java, Python, SQL; experience with Terraform, Kafka, Kinesis nice to have.
• Experienced in CI/CD, IaC, Git/SVN, Atlassian tools (JIRA, Confluence).
• Able to present technical issues to teams/Product Owners and manage/develop team members.

Culture & Benefits

• 10% on-target annual bonus.
• Market-leading pension (5% employee, 10% company), 26 days holiday + purchase scheme.
• Equalized parental leave (18 weeks full pay, 8 weeks half), paid carers leave up to 2 weeks.
• Online private GP 24/7, discounted BT/EE products, optional healthcare/dental/gym.
• Agile culture with openness, transparency, continuous improvement, and tech exploration opportunities.