Senior Databricks Data Engineer (AI)

TL;DR

Senior Databricks Data Engineer (AI): Building and optimizing data engineering solutions on Databricks with an accent on configuration, security, governance, and CI/CD for deployments. Focus on performance tuning Spark clusters, data modeling, and integrating AI model lifecycle management with MLFlow.

Location: 100% Remote, with occasional travel within Europe.

Company

hirify.global is a specialized IT consulting partner with 18 years of experience, driven by an experienced and senior team.

What you will do

• Build and optimize data engineering solutions using Databricks.
• Configure and manage Databricks Workspaces, including security controls, repos, and clusters.
• Develop and manage Databricks Notebooks (Python, SQL, PySpark), Jobs, Workflows, and Repos.
• Implement data security and governance using Unity Catalog, table/column-level security, and governed tags.
• Develop and maintain CI/CD pipelines for secure Databricks deployments.
• Integrate and manage AI model lifecycle using MLFlow for experiments access control and serving.

Requirements

• 5+ Years of Experience building Data engineering solutions.
• Proficiency in Databricks Workspaces, Notebooks (Python, SQL, PySpark), Jobs, Workflows, Repos, and Unity Catalog.
• Experience with Azure Data Platform Technologies including ADLS Gen2, Key Vault, Azure Monitor, Azure Log Analytics, and Azure Entra ID (AAD).
• Strong understanding of data security & governance practices, including Unity Catalog and secure credential handling.
• Experience with CI/CD for secure deployments (Azure DevOps or GitHub Actions) and source code version control (git).
• Knowledge of AI Engineering & Software Skills, including AI model lifecycle management and MLFlow.

Culture & Benefits

• No-term full-time contract.
• Health Insurance.
• 22 days of paid vacation plus 4 extra days annually (Carnival, Christmas Eve, New Year's Eve, Birthday).
• Meal Allowance on card (Coverflex).
• Referral Bonus and special discounts.
• Annual training budget.
• Team-oriented culture with a chance to grow in an international company.