Senior Data Engineer (Databricks/AWS)

Location: Europe

Company

Work through hirify.global on a project for a German-American deep-tech company developing direct-drive laser fusion technology and commercializing a laser fusion power plant.

What you will do

• Design and build scalable cloud-native data platforms from greenfield development through production.
• Implement near-real-time ingestion pipelines using event-driven patterns.
• Define and enforce data lake, lakehouse, medallion architecture, and data contract standards.
• Refactor and optimize Spark and PySpark scripts for performance and maintainability.
• Improve code quality, testing, CI/CD, data quality, observability, reliability, and cloud cost management.
• Build self-service tooling and microservices, adopt AI tooling and agentic workflows, and collaborate with scientific and engineering teams.

Requirements

• 5+ years of experience in data engineering and 3+ years of production experience with Databricks and Delta Lake.
• Strong expertise in medallion architecture and event-driven asynchronous pipeline design.
• Advanced proficiency in Python, PySpark, and SQL.
• Deep hands-on Databricks experience, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization.
• Strong AWS expertise, including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway.
• Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles, plus Git workflows, pull request reviews, and automated testing.
• Experience configuring secure CI pipelines using OIDC or service principals.
• Practical MLflow experience with experiment tracking, model registration, and reproducibility practices.
• Strong understanding of data quality, lineage, provenance, schema contracts, versioning, monitoring, observability, and cloud cost optimization.
• Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX.
• Upper-Intermediate English level or higher.
• Strong analytical mindset, ownership and accountability, clear communication of complex technical concepts, proactive problem-solving, attention to scalability and maintainability, and a collaborative team-oriented attitude.

Nice to have

• Experience packaging workloads in Docker containers and running batch jobs on AWS.
• Familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy.
• Experience building feature sets and train/test splits that avoid data leakage.
• Experience with regulated or high-security environments, including data residency requirements.
• Experience disabling or restricting AI features in secured environments.

Culture & Benefits

• Work on an ambitious deep-tech initiative focused on clean energy and commercial laser fusion.
• Contribute to projects connecting engineering with scientific research and sustainability.
• Collaborate directly with customer scientific and engineering teams.
• Drive engineering excellence, automation, and AI adoption within the data platform ecosystem.