Principal Software Engineer, Orchestration

Location: San Mateo, California, United States; office-based roles are onsite Tuesday, Wednesday, and Thursday, with optional presence on Monday and Friday.

Salary: $295,250–$345,040 USD annual base pay.

Company

Roblox builds a platform for developers and creators to deliver 3D immersive digital experiences to a global community.

What you will do

• Define the multi-year architecture and strategy for Apache Airflow and the surrounding data orchestration platform, including scheduler and executor design, multi-tenancy, upgrades, and migrations.
• Scale and harden Airflow across schedulers, metadata databases, executors, and DAG parsing to support thousands of daily workflows with predictable latency and strong SLAs.
• Design and operate AWS EKS clusters for orchestration and Spark workloads, covering autoscaling, resource isolation, cost efficiency, and deployment automation.
• Build APIs, custom operators and providers, libraries, and self-service tooling for safely developing, deploying, and monitoring data pipelines.
• Establish SLOs, alerting, and diagnostics; lead incident reviews and address root causes in code.
• Align roadmaps across data platform, governance, data science, and product engineering teams, while mentoring engineers and raising standards for design, code quality, and operational readiness.

Requirements

• Bachelor’s degree equivalent in computer science or sufficient equivalent experience.
• 10+ years of experience designing, building, and operating production-grade large-scale distributed systems.
• Deep hands-on experience designing, operating, and scaling a production data workflow orchestrator or job scheduler handling thousands of workflows or DAGs per day. Apache Airflow is preferred, while equivalent experience with Dagster, Prefect, Temporal, or a large in-house scheduler is accepted.
• Experience with scheduler and executor design, metadata store performance, task isolation, retries, idempotency, safe upgrades, and migrations.
• Production Kubernetes experience on AWS EKS, including Helm, Karpenter or Cluster Autoscaler, autoscaling, and GitOps-style deployment.
• Strong Python proficiency plus Go or Java, with experience writing clean, well-tested platform and tooling code.
• Working knowledge of Spark workloads on Kubernetes, including Spark on Kubernetes, spark-operator, or EMR on EKS, and their interaction with orchestration systems.
• Experience building platforms for other engineers, including APIs, observability, reliability, and developer experience.
• Experience influencing technical direction across a large engineering organization and driving multi-quarter projects from concept through stable production; strong ownership in ambiguous environments.

Nice to have

• Contributions to Apache Airflow or related open-source projects, including Airflow providers, Helm chart, spark-operator, Karpenter, or Argo CD.
• Experience with Dagster, Temporal, Argo Workflows, or Flyte, and the judgment to select an appropriate orchestration system.
• Experience with Terraform or other infrastructure-as-code tools, Prometheus or OpenTelemetry, and cost optimization or chargeback for shared platforms.
• Experience operating infrastructure at consumer-internet scale.

Culture & Benefits

• Full-time employees are eligible for equity compensation and benefits.
• Work in a large-scale environment serving more than 200 million daily active users.
• Reasonable accommodations are available during recruitment for qualifying disabilities or religious beliefs.
• Equal employment opportunities are provided without discrimination or harassment based on legally protected characteristics.