Data Engineer (Azure)

TL;DR

Data Engineer (Azure): Building and optimizing a Data Analytics Platform in Azure cloud with an accent on metadata-driven data pipelines and ETL optimization. Focus on developing reusable libraries, implementing Databricks Delta optimization, and integrating event-driven architectures using Kafka.

Location: Must be based in Poland (Remote or Hybrid in Krakow)

Company

An HR Tech company delivering innovative background screening solutions to Fortune 100 and Global 500 brands.

What you will do

• Develop reusable, metadata-driven data pipelines and shared ETL libraries.
• Automate and optimize data platform processes and resolve performance or quality issues in ETL processes.
• Build integrations with data sources and consumers using event brokers like Kafka, Event Hubs, or Kinesis.
• Develop and implement monitoring solutions for the Databricks data platform.
• Collaborate with infrastructure engineering to set up cloud resources and maintain platform documentation.
• Perform code reviews and initiate improvements to the data platform architecture.

Requirements

• Proficiency in Python, PySpark, and SQL.
• Experience building robust data pipelines using Databricks Spark and handling large, complex datasets.
• Knowledge of building data transformation modules as Python packages.
• Familiarity with Databricks Delta optimization techniques (partitioning, z-ordering, compaction).
• Experience developing CI/CD pipelines and working with event brokers.
• Must be based in Poland.

Nice to have

• Understanding of Spark Structured Streaming.
• Experience with Infrastructure as Code (Terraform, Bicep).
• Experience running containerized applications (Kubernetes, Azure Container Apps).
• Familiarity with change data capture (CDC) platforms like Debezium and event sourcing.
• Knowledge of Azure native solutions such as Azure Data Factory and Azure Function Apps.

Culture & Benefits

• Competitive benefits package including health care, life insurance, and Multisport.
• Modern and fully equipped office space in the heart of Krakow.
• Flexibility and possibility to work remotely.
• Opportunity to participate in a major technology transformation.
• Environment that encourages lifelong learning and personal development.