Lead Data Engineer (Databricks)

Location: Plano, TX, USA; full-time, onsite role.

Company

Brown & Brown provides insurance and related services and is expanding its Data & AI platform capabilities.

What you will do

• Lead the design, development, deployment, and operational support of scalable batch and streaming data solutions.
• Build Databricks-native solutions using Delta Lake, Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, and Unity Catalog.
• Develop production-grade ingestion, transformation, data-processing, feature engineering, and governed data products for AI, machine learning, analytics, semantic models, APIs, Genie Spaces, and AI agents.
• Implement data-quality validation, monitoring, observability, remediation, metadata, lineage, classification, certification, ownership, and access controls.
• Troubleshoot and optimize Spark workloads, operate production solutions, resolve incidents, manage migrations and upgrades, and improve performance and cost efficiency.
• Define engineering standards and reusable frameworks, lead design and code reviews, align architecture, remove technical blockers, and mentor engineers.

Requirements

• Bachelor’s degree in computer science, Engineering, Mathematics, Information Systems, or a related STEM field, or equivalent education and professional experience.
• 8+ years of experience in data engineering or related software/data engineering disciplines, including significant hands-on experience building and supporting production data solutions.
• Strong hands-on experience with Databricks, Apache Spark/PySpark, Delta Lake, Python, and SQL, including production-grade batch and streaming data pipelines and Databricks capabilities such as Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, and Unity Catalog.
• Experience designing reliable workflow orchestration with scheduling, dependencies, parameters, retries, notifications, and operational monitoring.
• Experience implementing Unity Catalog governance and access controls, including catalogs, schemas, tables, views, external locations, storage credentials, lineage, and fine-grained permissions.
• Experience with Terraform, Databricks Asset Bundles, or comparable Infrastructure as Code approaches, together with Git and CI/CD practices.
• Strong experience troubleshooting and optimizing Spark workloads using Spark UI, job logs, query plans, partitioning, join optimization, and Adaptive Query Execution.
• Strong understanding of conceptual, logical, relational, dimensional, and physical data modeling, plus experience delivering ETL/ELT solutions through deployment and production support.
• Demonstrated ability to balance engineering quality, delivery speed, scalability, resiliency, and cost; diagnose complex data and production issues; lead solution design, code reviews, standards, mentoring, and team unblocking; influence technical direction and build consensus without formal authority.
• Experience working within Agile delivery models such as Scrum, Kanban, or SAFe.

Nice to have

• Experience as a technical lead, lead engineer, or senior engineer with responsibility for technical design and engineering standards.
• Advanced Python or Scala; cloud platforms including Microsoft Azure, AWS, or Google Cloud; and cloud storage such as Azure Data Lake Storage Gen2, Amazon S3, OneLake, or Google Cloud Storage.
• Experience with Snowflake, Apache Iceberg, Apache Hudi, BigQuery, messaging and event-streaming technologies, Cassandra, HBase, REST APIs, or web services.
• Experience developing governed data products for AI, Generative AI, machine learning, semantic models, APIs, advanced analytics, Genie Spaces, or AI agents, including enterprise data governance, data quality, metadata, and lineage.
• Experience with insurance or health data and AI-assisted development tools such as GitHub Copilot, Claude, or Databricks AI-assisted development capabilities.
• Deep Databricks Lakehouse experience, high-volume or near-real-time ingestion, reusable Python frameworks and platform capabilities, and a track record of simplifying architectures and raising engineering standards.

Culture & Benefits

• Meritocratic culture emphasizing self-starters, customer focus, collaboration, continuous learning, accountability, and engineering excellence.
• Medical, prescription, dental, vision, life, and disability insurance.
• Employee stock purchase plan, 401(k), student loan assistance, and tuition reimbursement.
• Mental health and enhanced advocacy services, paid time off, holidays, and preferred partner discounts.