Data Engineering Lead (Azure Databricks)

TL;DR

Data Engineering Lead (Azure Databricks): Designing and implementing scalable end-to-end data pipelines and integrating complex ML models into robust ETL systems with an accent on Medallion Architecture and cost-efficiency. Focus on optimizing Apache Spark jobs, architecting high-performance PostgreSQL schemas, and leading a lean engineering team through technical stewardship.

Location: Remote (USA)

Company

Industry leader in marketing intelligence providing a next-generation platform for media, creative, and business strategy insights.

What you will do

• Architect and implement complex end-to-end data pipelines using Azure Databricks and PySpark.
• Design and maintain a scalable data architecture using the Medallion Architecture (Bronze/Silver/Gold layers).
• Optimize Apache Spark jobs and cluster policies to minimize cloud compute costs and refactor pipelines periodically.
• Develop a proactive monitoring framework and a Data Validation Framework to ensure 99.9% reliability and accuracy.
• Architect high-performance PostgreSQL schemas, managing indexing, partitioning, and complex analytical queries.
• Lead a lean team through sprint cycles and code reviews, promoting cross-trained agility and CI/CD best practices.

Requirements

• 10+ years of experience in Data or Software Engineering with deep codebase involvement.
• 3+ years of experience as a Technical Lead managing agile teams.
• Expertise in Python, PostgreSQL, pgvector, Azure Databricks, PySpark, and Delta Lake.
• Experience with Docker, Git, Azure DevOps, and CI/CD.
• Must be based in the USA.

Nice to have

• Experience with Apache Airflow for workflow orchestration.
• Familiarity with Azure Kubernetes Service (AKS).

Culture & Benefits

• High-velocity "player-coach" environment with 70-80% hands-on coding.
• Cultural focus on cross-trained agility and moving away from siloed specializations.
• Opportunity to work on a high-impact, lean team managing massive data volumes from diverse media channels.