Lead Data Engineer (Databricks, Azure, PySpark)

TL;DR

Lead Data Engineer (Databricks, Azure, PySpark): Design, build, and optimize scalable data pipelines using PySpark and Databricks with an accent on production-grade data processing, code quality, and automation. Focus on high-performance orchestration, CI/CD deployments in Azure DevOps, and mentoring engineers to raise engineering standards.

Company

hirify.global is a tech recruitment agency connecting data engineering talent with organizations across Europe.

What you will do

• Design, build, and optimize scalable data pipelines using PySpark and Databricks for large-scale data processing.
• Develop and maintain high-quality Python code using engineering best practices (linting, formatting, type hinting).
• Contribute to automated testing frameworks using pytest to ensure reliability and maintainability.
• Support and enhance CI/CD pipelines within Azure DevOps for robust, repeatable deployments.
• Collaborate with cross-functional teams to drive technical decisions, improve engineering standards, and mentor junior and medior Data Engineers.

Requirements

• Around 10 years of experience in Data Engineering delivering production-grade data platforms.
• Hands-on expertise in Databricks, PySpark, and Python for large-scale data processing and orchestration.
• Strong experience with Azure Data Factory, Azure DevOps, CI/CD practices, and automated testing frameworks.
• Strong software engineering best practices: code quality, peer reviews, and maintainable solutions.
• Proven ability to mentor engineers and influence technical direction.

Culture & Benefits

• Engineering excellence with a focus on automation and data quality.
• Continuous improvement and collaboration across teams.
• Emphasis on maintainable solutions, code quality, and peer reviews.
• Mentoring and shared technical ownership to raise engineering standards.

Hiring process

• Chat after confirming the role requirements.

Location: Amsterdam, North Holland

Salary: Negotiable EUR / Year