Senior Principal Engineer (Kafka)

Location: Remote within the United States; listed locations include Poughkeepsie, Lowell, Rochester, Tucson, Research Triangle Park, Armonk, Boston, Bellevue, Atlanta, San Jose, Dallas, Austin, San Francisco, Seattle, and the states of Georgia, Minnesota, Texas, New York, North Carolina, Arizona, Washington, Massachusetts, and California. Some travel may be required based on business demand.

Salary: $192,000–$358,000 per year.

Company

IBM Software develops hybrid cloud, AI, and data platforms, while the role supports Confluent’s real-time data streaming platform and Kafka architecture.

What you will do

• Define the long-term technical strategy, architecture, roadmap, and investment priorities for the Kafka Background Plane.
• Design and optimize distributed scheduling, task coordination, isolation, autoscaling, backpressure, and dependency-execution workflows.
• Own correctness for compaction, log rewriting, retention, segment lifecycle, metadata workflows, and asynchronous object garbage collection under concurrency and partial failures.
• Scale background processing across partitions, throughput, storage, and lag while preserving sub-millisecond foreground latency.
• Lead cross-organizational architecture decisions covering data references, metadata interfaces, object lifecycles, encryption, and task scheduling through production rollout.
• Establish fault-tolerance, chaos-testing, observability, canary-safety, incident-mitigation, mentoring, and technical leadership practices.

Requirements

• Bachelor’s degree required.
• Proven experience architecting, building, and operating mission-critical distributed systems at planetary scale, with deep expertise in storage engines, distributed log stores, stream processing, replication, or consensus protocols.
• Deep knowledge of Apache Kafka internals, including storage semantics, log compaction, tiered storage, and segment indexing, or equivalent high-throughput streaming systems.
• Experience defining and delivering complex multi-year architectural roadmaps across multidisciplinary engineering organizations.
• Exceptional ability to reason about race conditions, distributed concurrency, state-machine transitions, edge-case failures, and backpressure.
• Experience establishing measurable SLA/SLO metrics, durability targets, and automated recovery practices for global multi-tenant infrastructure.
• Outstanding written and oral communication skills, including executive-level technical communication and cross-team alignment.
• Commitment to mentoring, talent development, and enabling staff and principal engineers.

Nice to have

• Master’s degree.
• Experience with log-structured storage formats, custom storage engines, cloud object stores, or large-scale data-rewriting pipelines.
• Experience designing fault-tolerant object lifecycle strategies that prevent data leakage, orphaned resources, and premature deletion.
• Experience with chaos engineering, production traffic replay, deterministic simulation testing, and large-scale load generation.
• Experience with multi-cloud, multi-region deployments across AWS, GCP, and Azure.

Culture & Benefits

• Remote work arrangement with possible business travel.
• Healthcare, dental, vision, mental health, life insurance, disability coverage, and financial programs including 401(k) and pension benefits.
• Paid holidays, sick time, vacation, parental bonding leave, and other paid care leave.
• AI-driven learning resources, training, and professional certification opportunities.
• Inclusive employee resource groups, volunteer opportunities, and retail and service discounts.