Principal Data Engineer (AI)

TL;DR

Principal Data Engineer (AI): Building and enhancing an agentic data platform for LLM-powered research systems with an accent on designing scalable applications, creating efficient data pipelines, and generating embeddings. Focus on utilizing Python, Java, Spark, and AWS for robust, intelligent solutions.

Location: London. Offers home, office, or commuting allowance with flexible working hours.

Company

hirify.global is a global leader in information and analytics, empowering researchers and healthcare professionals through quality information and vast data sets combined with analytics.

What you will do

• Design, prototype, and build robust, scalable applications following best practices.
• Work with Python, Java (Spring Boot), Spark, Airflow, and various database technologies.
• Create scalable and optimized ELT/ETL data pipelines for bulk data ingestion.
• Utilize data engineering and warehousing technologies like Data Lake and Delta Lake.
• Generate embeddings for ingestion with the latest models at scale.
• Build and automate cloud infrastructure in AWS for service hosting and monitoring.

Requirements

• Strong commercial software engineering experience delivering server-side applications.
• Good experience with ETL, Spark, and Airflow.
• Experience with Database systems.
• Excellent programming skills in Java and Python.
• Advocacy for best practices including clean code, architectural design patterns, and CI/CD.
• Comfortable learning new technologies to keep products current in a rapidly evolving ecosystem.
• Inclusive, positive, and collaborative mindset with a desire to deliver business value.

Culture & Benefits

• Healthy work/life balance with flexible working hours.
• Comprehensive Pension Plan.
• Generous vacation entitlement and option for sabbatical leave.
• Maternity, Paternity, Adoption, and Family Care leave.
• Personal Choice budget and various employee discounts.
• Internal communities and networks, Employee Assistance Program.