Senior Data Engineer (Databricks)

TL;DR

Senior Data Engineer (Databricks): Building and migrating large-scale batch data pipelines into Databricks with an accent on production-grade Python/Spark/SQL development, migration validation, and pipeline reliability. Focus on leading technical design discussions while solving complex performance, testing, monitoring, and troubleshooting challenges during legacy-to-cloud data transitions.

Location: United States

Company

Modern product engineering partner helping organizations modernize data ecosystems and accelerate AI.

What you will do

• Build and migrate large-scale batch data pipelines from legacy platforms into Databricks.
• Develop and optimize production-grade solutions using Python, Spark, and advanced SQL.
• Support parallel legacy and cloud data environments during migration and validation.
• Lead technical design conversations and guide implementation within an established architecture while staying hands-on with development and troubleshooting.
• Improve pipeline reliability and performance using testing, monitoring, tuning, and production support (including rotating on-call).
• Use AI tools (including Claude) and GitHub Copilot to streamline work and share best practices.

Requirements

• Recent hands-on experience building production batch solutions in Databricks (ideally within the past 12–18 months).
• Strong development experience with Python and Apache Spark.
• Advanced SQL skills, including complex query development and performance tuning.
• Experience delivering large-scale migrations from legacy data platforms to cloud environments (preferably involving Teradata or a comparable platform) and supporting parallel legacy/cloud systems during migration.
• Ability to lead design/technical-direction discussions while actively coding and implementing.
• Must be authorized to work in the U.S. without sponsorship.

Culture & Benefits

• Remote or hybrid work options.
• Matching 401(k) plan, substantial PTO, and robust benefits package.
• Ongoing growth via training, mentorship, and industry certifications.
• Tuition reimbursement.
• Opportunity to work with cutting-edge technologies and participate in an AI Innovators Program.

Hiring process

• Interviews focused on technical experience with Databricks/Python/Spark/SQL and migration work.
• Discussion of collaboration style, communication, and ability to lead design while staying hands-on.
• On-call rotation expectations reviewed during the process.