Senior Data Engineer (Databricks)

TL;DR

Senior Data Engineer (Databricks/PySpark): Designing and optimizing next-generation Lakehouse data platforms with an accent on transforming complex legacy datasets into high-quality data products. Focus on implementing scalable ingestion pipelines, Delta Lake principles, and integrating AI agents for code stability.

Location: Remote (Bulgaria)

Company

hirify.global is an IT services company providing technical expertise and diverse career growth opportunities across various projects.

What you will do

• Design and implement scalable data ingestion and transformation pipelines using Databricks, PySpark, and Airflow.
• Implement Delta Lake principles, focusing on CDC, schema evolution, and data quality frameworks (e.g., Great Expectations).
• Develop and optimize complex SQL and Python scripts to integrate diverse data sources including APIs and S3 files.
• Support the implementation of data governance and cataloguing solutions such as Unity Catalog.
• Guide and manage AI agents for code generation, ensuring quality and stability of the output.
• Collaborate proactively to drive technical solutions in a fast-paced rebuild environment.

Requirements

• 5+ years of professional experience in data engineering, focusing on cloud and distributed processing environments.
• Strong experience with Databricks, PySpark, and Delta Lake, with deep knowledge of CDC and schema evolution.
• Expert SQL optimization and Python skills, with hands-on experience using Airflow or similar tools.
• Familiarity with AWS components and a good understanding of CI/CD for data workflows.
• Knowledge of streaming platforms (Kafka/Kinesis) and exposure to Unity Catalog or similar governance tools.
• Ability to work independently and direct AI agents while maintaining high quality standards.

Culture & Benefits

• Internal Mobility and Referral programs for career growth and job rotations.
• hirify.global Club celebrating long-term cooperation anniversaries with gifts.
• CSR events, team buildings, and support for industry events as a speaker or visitor.
• Education benefits, including learning courses, certifications, and language courses.
• Multisport membership and technical/personal training.
• For employment contracts: Cafeteria, financial contribution for meals, life insurance, and bonuses for personal events.