Lead Data Engineer (Databricks)

Location: London or Eastleigh, UK, with office attendance twice a month. Flexible working is available within the UK.

Salary: £95,000–£105,000 per year.

Company

Large UK car and home insurance provider serving more than four million customers.

What you will do

• Lead the development and continuous improvement of a modern cloud-based data platform and engineering standards.
• Design, build, and maintain scalable data pipelines and trusted data products for reporting, analytics, modelling, machine learning, and AI.
• Mentor and support data engineers, promoting knowledge sharing and continuous improvement.
• Improve platform reliability, performance, security, and cost efficiency.
• Drive best practices in testing, automation, CI/CD, and infrastructure as code.
• Work with analytics, AI, and business teams to solve data challenges and shape data strategy.

Requirements

• Strong experience in data engineering and cloud-based data platforms.
• Hands-on experience with Databricks, Spark, SQL, and Python.
• Understanding of Airflow, Terraform, and modern engineering practices.
• Experience building large-scale batch and real-time data pipelines.
• Strong problem-solving, decision-making, communication, and collaboration skills.
• Interest in AI and emerging technologies, with a focus on quality and continuous improvement.

Culture & Benefits

• Permanent employment with flexible full-time, part-time, and job-share arrangements.
• Minimum of 35 days of holiday including bank holidays, with the option to buy and sell days.
• Health, dental, critical illness, mental health, and wellbeing support.
• Annual bonus schemes, salary reviews, pension, employee savings, and loans.
• Family leave, partner life assurance, return-to-work support, and employee social activities.
• Flexible working arrangements that support working from home within the UK.