Senior Data Engineer (Databricks)

TL;DR

Senior Data Engineer (Databricks): Designing and optimizing scalable data pipelines and processing algorithms using the Databricks Lakehouse Platform with an accent on Apache Spark, Delta Lake, and Unity Catalog. Focus on building production-grade batch and streaming workflows to power analytics and machine learning use cases.

Location: Must be based in Spain (Remote or Hybrid)

Company

A leading international Data Engineering consultancy specializing in Big Data, Cloud, Streaming, IIoT, and Microservices.

What you will do

• Design, implement, and maintain scalable data pipelines using Databricks Lakehouse, Apache Spark, Delta Lake, and Unity Catalog.
• Lead the development of batch and streaming data workflows for analytics, machine learning, and BI use cases.
• Collaborate with data scientists, architects, and stakeholders to translate complex requirements into production-grade solutions.
• Optimize performance and cost-efficiency of Databricks clusters using Photon, Auto Loader, and Job Workflows.
• Establish and enforce best practices for data quality, governance, and security.
• Mentor junior engineers and contribute to the team's Databricks expertise.

Requirements

• Deep hands-on experience with Databricks on Azure, AWS, or GCP, including Spark (PySpark/Scala), Delta Lake, and MLflow.
• Strong programming skills in Python or Scala and experience with CI/CD pipelines.
• Solid understanding of distributed computing, data modeling, and performance tuning in cloud environments.
• Familiarity with orchestration tools like Airflow and infrastructure-as-code via Terraform.
• Professional English and Spanish communication skills (C1-level).
• Must be located in Spain.

Culture & Benefits

• Flexible work options, including fully remote or hybrid arrangements.
• Professional development with paid certifications and unlimited access to Udemy Business.
• Individual development plans, annual performance reviews, and regular feedback cycles.
• Non-hierarchical environment focusing on knowledge-sharing and networking.
• Local, virtual, and global team events to foster community.
• Strong emphasis on work-life balance.