Staff Software Engineer (Streaming)

TL;DR

Staff Software Engineer (Streaming): Building and optimizing the Spark Structured Streaming engine for the Databricks Data Intelligence Platform with an accent on state management, latency, and throughput. Focus on re-imagining engine architecture, designing new operators, and driving stream processing adoption across the product portfolio.

Location: Seattle, Washington

Salary: $182,400 — $247,000 USD

Company

Databricks is the data and AI company providing a unified platform for data, analytics, and AI used by over 10,000 organizations worldwide.

What you will do

• Set and drive the technical vision for Spark Structured Streaming across OSS and the Databricks Data Intelligence Platform.
• Design and build core engine capabilities, including advanced state management and new operators.
• Improve engine architecture to optimize latency, throughput, and cost.
• Raise the bar for engineering quality and operability to ensure software is production-ready.
• Drive stream processing adoption across the entire Databricks product portfolio.
• Mentor and technically lead engineers while partnering with management to grow talent.

Requirements

• BS or higher in Computer Science, a related technical field, or equivalent practical experience.
• 8+ years of experience building big-data ecosystems, Apache Spark, or database internals.
• Deep passion for distributed systems, storage systems, language design, or performance optimization.
• Proven track record of delivering features while maintaining high operational excellence.
• Ability to work cross-functionally with product management and customers to understand user personas.
• Must be based in Seattle, Washington.

Culture & Benefits

• Comprehensive benefits and perks tailored to the employee's region.
• Opportunity to contribute to high-impact open-source technology like Apache Spark.
• Inclusive culture committed to diversity, equity, and equal employment opportunity.
• Collaborative environment focused on solving the world's toughest data and AI problems.