Senior Data Engineer (Databricks)

Location: Madrid is required; remote work or work from one of the offices is available.

Company

hirify.global delivers software engineering and data solutions for business stakeholders, product teams, and external vendors.

What you will do

• Design and implement complex batch and streaming ETL/ELT pipelines with Python, SQL, and Spark for massive datasets.
• Build scalable, secure, and cost-effective data solutions using AWS Data Analytics services, Databricks, and lakehouse architectures.
• Manage orchestration and deployment with Airflow, Docker, and ECS, including repeatable infrastructure through CloudFormation or Terraform.
• Monitor and optimize Spark cluster performance, AWS spend, storage strategies, availability, and latency.
• Implement IAM roles, bucket policies, KMS encryption, and other security controls for data at rest and in transit.
• Collaborate in an Agile environment with data scientists, product and application managers, AI and analytics teams, business stakeholders, and vendors; lead up to 2 direct reports and coordinate around 15 external contributors.

Requirements

• 4+ years of hands-on experience in active Big Data environments and 2+ years specializing in Data Analytics within AWS.
• Experience architecting and managing Spark clusters with Amazon EMR; developing AWS Glue serverless ETL jobs, Data Catalogs, and Glue Crawlers.
• Experience with Amazon S3 data lakes, including partitioning, Parquet or Avro compression, and lifecycle policies; Amazon Redshift schemas and query optimization; and Amazon Athena ad-hoc SQL analysis.
• Experience with open table formats such as Iceberg or Delta, Databricks, data mesh architectures, and Lakehouse bronze, silver, and gold layers using Delta Lake.
• Experience deploying and scaling Amazon MWAA environments, using AWS Lambda for event-driven triggers and microservices, and using Airflow for orchestration.
• Expert-level proficiency in Spark, Python, and SQL, plus experience with Docker and ECS.
• Knowledge of Git, CI/CD for data pipelines, automated testing, code quality, and documentation.
• Degree in Computer Science, Engineering, Mathematics, or a related field, or equivalent practical experience.
• Excellent written and oral English communication skills and the ability to explain complex technical concepts to non-technical audiences; Madrid location is required.

Nice to have

• Experience with Amazon Kinesis or MSK for real-time ingestion, and with Flink or Kafka distributed messaging applications.
• Java programming experience and experience industrialising ML use cases.
• Experience with QlikView or QlikSense for BI initiatives.
• Experience in fast-paced Scrum or Kanban environments.
• AWS Certified Data Engineer, AWS Certified Solutions Architect, or Databricks Data Engineer certification.
• Experience with OpenShift, GitHub Actions, or Jenkins for CI/CD of data workflows.

Culture & Benefits

• Remote work or work from one of the company offices with a flexible working schedule.
• 20 business days of covered vacation and 5 additional days off.
• Free English classes.
• Friendly and supportive work atmosphere.
• Medical insurance for employees from Ukraine and legal support.