обновлено 3 дня назад
Staff Data Platform Engineer (AI)
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Описание вакансии
Текст:
TL;DR
Staff Data Platform Engineer (AI): Building a globally distributed Data Transfer Hub for ingesting video, LiDAR, and sensor data from partners at up to petabytes-per-day scale with an accent on reliability, throughput, observability, and cloud-to-cloud networking. Focus on designing parallelised transfer pipelines with Flyte, Kafka, Azure, Python, and Terraform while leading technical delivery in a fast-changing environment.
Location: London, United Kingdom; full-time role based in the London office with a hybrid working policy.
Company
develops embodied AI software and foundation models for automated driving, including mapless and hardware-agnostic autonomy products for automakers.
What you will do
- Set the technical direction for the Data Transfer Hub, balancing reliability, throughput, cost, partner constraints, observability, and operational support.
- Lead the technical design and delivery of a globally distributed hub-and-spoke ingestion system for video, LiDAR, and sensor data at up to petabytes-per-day scale.
- Design and build parallelised data transfer pipelines using Flyte, Kafka event-driven patterns, and Azure storage and transfer infrastructure.
- Build and operate cloud-to-cloud and cross-cloud networking solutions for large-scale data movement.
- Write infrastructure-as-code with Terraform and develop pipeline logic primarily in Python.
- Work directly with external data partners, including Uber and Nissan, while adapting to evolving requirements.
Requirements
- Experience building and operating large-scale data transfer pipelines at multi-terabyte or petabyte scale.
- Experience orchestrating parallelised transfer jobs; Flyte is preferred, while Airflow and Prefect are also considered.
- Experience building reliable and observable data systems with retries, backfills, data integrity checks, lifecycle tracking, and operational alerting.
- Strong cloud platform skills, with Azure preferred, plus cloud-to-cloud networking and large-scale blob or object storage experience.
- Proficiency in Python and Terraform.
- Product mindset, autonomy, ownership, and comfort working with ambiguity and evolving requirements.
Nice to have
- Experience with Kafka for event-driven transfer observability.
- Multi-cloud experience across Azure, AWS, and/or GCP.
- Experience working directly with external data partners.
- Background in autonomous vehicles, robotics, or other data-intensive domains.
Culture & Benefits
- Hybrid work combining time in offices and workshops with time working from home.
- Core working hours with flexibility to determine a schedule that works for the team.
- Fast-paced environment with a startup-style team within a scale-up.
- Inclusive, diverse, and respectful workplace focused on learning, collaboration, and impact.
Hiring process
- Inclusive interview experience with accommodations or adjustments available on request.
- Application information can be prefilled using the candidate's CV.
Будьте осторожны: если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого - это мошенники. Обязательно жмите "Пожаловаться" или пишите в поддержку. Подробнее в гайде →
Похожие вакансии
6 дней назад
Data Engineer (Azure)
7 дней назад
Senior Data Platforms Engineer - Enterprise Data Platforms
7 дней назад
Backend Software Engineer - Application Development
CoreWeave
5 дней назад
Senior Software Engineer (Physical AI)
98 000 - 130 000GBP
CoreWeave
5 дней назад
Staff Software Engineer (Physical AI)
116 000 - 155 000GBP
7 дней назад