Azure Senior Data Engineer (Python/PySpark)

TL;DR

Senior Data Engineer (Azure): Building and optimizing robust, metadata-driven data pipelines and automation solutions within the Azure cloud platform with an accent on Databricks Spark, ETL libraries, and performance optimization. Focus on resolving quality issues, improving data platform architecture, and integrating with diverse data sources.

Location: Remote from Poland, Latin America, Moldova, or Romania

Company

hirify.global is an outsourcing company providing solutions for clients, including one specializing in risk management and talent acquisition.

What you will do

• Develop reusable, metadata-driven data pipelines.
• Automate and optimize data platform processes.
• Build integrations with data sources and consumers.
• Add data transformation methods to shared ETL libraries.
• Contribute to data platform wiki/documentation and perform code reviews.
• Initiate and implement improvements to data platform architecture.

Requirements

• Programming: Python/PySpark, SQL required.
• Proficient in building robust data pipelines using Databricks Spark.
• Experience with large, complex datasets and building data transformation modules.
• Familiarity with Databricks Delta optimization techniques.
• Experience in developing CI/CD pipelines.
• Experience leveraging event brokers (Kafka /Event Hubs / Kinesis).
• Familiarity with Agile Software Development methodologies (Scrum).

Nice to have

• Understanding of stream processing challenges and Spark Structured Streaming.
• Experience with IaC (Terraform, Bicep) and containerized applications (Azure Container Apps, Kubernetes).
• Experience building event sourcing solutions.
• Knowledge of Azure cloud native solutions (Azure Data Factory, Azure Function App, Azure Container Instances).

Culture & Benefits

• Flexible employment and remote work.
• Work with latest technologies and international projects.
• Non-corporate atmosphere.
• Language classes and internal/external training.
• Private healthcare and insurance.
• Monthly sport budget (Multisport card) and well-being initiatives.