Senior Data Engineer (AI)

TL;DR

Senior Data Engineer (AI): Designing, building, and maintaining robust data pipelines and infrastructure for large-scale AI models with an accent on scalability, distributed systems, and real-time data processing. Focus on implementing efficient orchestration frameworks, optimizing cloud-native environments, and ensuring high data integrity across the platform.

Location: Kraków, Poland (Onsite)

Company

hirify.global is an AI-driven company developing Large Market Models (LMM) for demand forecasting and algorithmic decision-making across various industries.

What you will do

• Design and maintain scalable ETL/ELT pipelines for complex data workflows.
• Implement orchestration frameworks to manage data task dependencies and execution.
• Optimize data infrastructure for performance and reliability within distributed systems.
• Deploy and manage cloud-native, containerized environments using Docker and Kubernetes.
• Develop and enforce CI/CD processes and automated data testing frameworks.
• Collaborate with cross-functional teams to align data architecture with business and research needs.

Requirements

• 5+ years of professional experience with Python and SQL.
• Proven track record of building and managing batch and streaming data pipelines.
• Experience with orchestration tools like Airflow or Dagster.
• Hands-on experience with distributed computing frameworks such as Spark, Dask, or Beam.
• Proficiency in cloud platforms and containerization (Docker/Kubernetes).
• Strong background in software engineering best practices including TDD and code quality.

Nice to have

• Experience with dbt, ClickHouse, or Ray.
• Familiarity with the Python data stack (pandas, NumPy, Apache Arrow).
• Background in functional programming.

Culture & Benefits

• Work in a cutting-edge environment focused on deep learning and AI-driven algorithmic trading.
• Engage with large-scale data challenges and modern technology stacks.
• Collaborate within a specialized data engineering and research-focused team.
• Opportunity to influence architecture in a rapidly expanding AI company.