Senior Data Engineer (Databricks)

TL;DR

Senior Data Engineer (Databricks/Spark): Designing, building, and scaling a modern data platform for analytics and data products across media brands with an accent on lakehouse architecture and real-time ingestion. Focus on implementing medallion pipelines, optimizing Spark performance, and ensuring data governance through Unity Catalog.

Location: Hybrid: Must be based in New York City (minimum 3 days per week in office)

Salary: $140,000 - $160,000 yearly

Company

An industry-leading media and entertainment business (Nasdaq: VSNT) managing iconic brands such as CNBC, USA Network, and Golf Channel.

What you will do

• Design and implement Lakehouse architecture using Delta Lake, including medallion pipeline patterns (Bronze/Silver/Gold).
• Build and operate batch and real-time ingestion pipelines leveraging Databricks Auto Loader, Structured Streaming, and CDC.
• Implement data governance and security using Unity Catalog and RBAC for sensitive environments.
• Optimize system performance and manage costs through Spark tuning, cluster sizing, and FinOps strategies.
• Design and maintain CI/CD pipelines and orchestration workflows using Databricks Workflows and Airflow.
• Collaborate with Data Science teams on ML workflows, including MLflow and feature store integration.

Requirements

• Bachelor’s degree in Computer Science, Data Engineering, or equivalent practical experience.
• 5+ years of experience building production-grade data pipelines in cloud environments using Spark-based platforms.
• Expertise in PySpark, SQL, and distributed processing frameworks (e.g., Spark, Flink).
• Proficiency with orchestration tools such as Apache Airflow or Dagster.
• Strong experience with Git and collaborative development workflows.
• Must be able to attend in-person interviews in New York.

Nice to have

• Experience with Lakeflow Spark Declarative Pipelines (SDP) and MLOps practices.
• Background in media and entertainment data, such as ad tech and audience analytics.
• Databricks or cloud certifications (e.g., Databricks Certified Data Engineer).
• Comfortable using AI-assisted development tools like ChatGPT.

Culture & Benefits

• Comprehensive company-sponsored health insurance (medical, dental, and vision).
• 401(k) retirement plan and paid leave.
• Tuition reimbursement for professional development.
• Collaborative environment with a focus on technical mentorship and growth.