Databricks Data Engineer (Public Sector)

TL;DR

Databricks Data Engineer (Public Sector): Develop and maintain scalable data pipelines and ETL processes using Databricks and Apache Spark to consolidate datasets into Salesforce solutions with an accent on data integration, governance, and performance optimization. Focus on designing unified data architectures, enabling advanced analytics, and ensuring data quality and security in a government-focused environment.

Location: Client sites, remote, or hybrid depending on project need, based in London, United Kingdom

Salary: £500 - £550 per day

Company

hirify.global Plc is a digital, data, and solutions consultancy with over 35 years’ experience serving primarily the UK public sector and central government.

What you will do

• Design, develop, and maintain data pipelines using Databricks and Apache Spark.
• Implement ETL/ELT processes for structured and unstructured data.
• Optimize data workflows for performance and scalability.
• Collaborate with data scientists and analysts to support advanced analytics and ML models.
• Ensure data quality, governance, and security across platforms.
• Integrate data from multiple sources into a unified architecture and monitor pipeline reliability.

Requirements

• Must have SC clearance or equivalent security clearance.
• Proven experience as a Data Engineer with strong skills in Databricks, Apache Spark, and PySpark.
• Hands-on experience with cloud platforms such as Azure, AWS, or GCP.
• Solid understanding of data warehousing, SQL, and data modelling.
• Knowledge of data governance, security, and compliance standards.
• Excellent problem-solving and communication skills.

Nice to have

• Experience with Azure or AWS Databricks.
• Familiarity with MLflow, Airflow, or other orchestration tools.
• Exposure to machine learning workflows and data science collaboration.
• Relevant Databricks certifications.

Culture & Benefits

• Supportive and collaborative culture with knowledge sharing.
• Opportunities to work on national-scale projects impacting public safety and environment.
• Encouragement of creativity, autonomy, and innovation.
• High employee satisfaction and CEO approval ratings.