Senior Data Engineer (Databricks)

TL;DR

Senior Data Engineer (Databricks): Design, develop, and optimize data pipelines and workflows using Databricks and AWS with an accent on PySpark, Spark SQL, Delta Lake, and performance tuning. Focus on building reliable, secure, and compliant data models and transformations, implementing monitoring/logging, and resolving complex production pipeline issues.

Location: London, UK

Company

hirify.global provides asset servicing and operational solutions for investment management, including fund administration and related services.

What you will do

• Design, develop, and maintain high-performance data pipelines and workflows in Databricks.
• Collaborate with architects and analysts to build data models and transformations.
• Ensure data quality, security, and compliance within Databricks environments.
• Implement monitoring and logging to improve pipeline reliability and performance.
• Deploy and manage code via CI/CD workflows.
• Troubleshoot and resolve complex data processing and production pipeline issues (supporting L2/L3).

Requirements

• Proficiency in Databricks, including PySpark, Spark SQL, and Delta Lake.
• Strong experience with data modeling, optimization, and performance tuning within Databricks.
• Hands-on AWS experience for Databricks deployments, including AWS DMS and S3-based ingestion patterns.
• Solid understanding of data governance, security best practices, and compliance.
• Familiarity with CI/CD principles and practices.
• Excellent problem-solving skills and ability to collaborate effectively in agile teams.

Nice to have

• Databricks certification (Data Engineer Professional).
• Experience working in financial services or other highly regulated environments.
• Experience supporting and troubleshooting data pipelines.

Culture & Benefits

• Hybrid working model.
• Opportunity to impact and shape the strategic direction of the data platform.
• Professional growth opportunities and competitive compensation.
• Connected team with a focus on innovation, client focus, and learning & development.

Hiring process

• Application review followed by interviews for selected candidates.
• Only candidates proceeding to the interview stage will be contacted.