Data Engineer (Databricks)

Location: Plano, TX, USA

Company

Brown & Brown provides insurance and related services and is hiring for its Data & AI Platform team.

What you will do

• Design, develop, test, deploy, monitor, troubleshoot, and optimize scalable batch and streaming data solutions.
• Build Databricks-native Lakehouse solutions using Delta Lake, Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, and Unity Catalog.
• Develop production-grade ingestion, transformation, ETL/ELT, orchestration, data-quality, monitoring, observability, and remediation capabilities.
• Create governed data assets and products supporting AI, Machine Learning, Generative AI, analytics, semantic models, APIs, Genie Spaces, and AI agents.
• Optimize Spark workloads, improve reliability and cost efficiency, support incidents and migrations, and maintain reusable engineering frameworks and documentation.
• Collaborate with product owners, architects, operations teams, business stakeholders, and other engineers through technical design, code review, testing, and continuous improvement.

Requirements

• Bachelor’s degree in computer science, engineering, mathematics, information systems, or a related STEM field, or equivalent education and professional experience.
• At least 4 years of experience in data engineering, software engineering, or a related technical discipline, including production data solutions.
• Hands-on experience with Apache Spark/PySpark, Delta Lake, Python, SQL, batch and/or streaming pipelines, ETL/ELT, and Databricks capabilities including Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, and Unity Catalog.
• Experience with workflow orchestration, scheduling, dependencies, parameters, retries, notifications, monitoring, automated testing, deployment pipelines, and production support.
• Understanding of Unity Catalog and data governance, including catalogs, schemas, tables, views, lineage, access controls, permissions, metadata, classification, certification, and ownership.
• Strong Python and SQL development skills, including reusable frameworks, libraries, utilities, maintainable and testable production solutions, and AI-assisted development tools such as GitHub Copilot, Claude, or Databricks AI-assisted development capabilities.
• Experience with Git, CI/CD, Infrastructure as Code such as Terraform or Databricks Asset Bundles, Agile delivery models including Scrum, Kanban, or SAFe, and the full engineering lifecycle.
• Experience troubleshooting and optimizing Spark workloads using Spark UI, job logs, query plans, partitioning, join optimization, and Adaptive Query Execution; strong problem-solving, analytical, written, and verbal communication skills.
• Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud; cloud storage such as Azure Data Lake Storage Gen2, Amazon S3, OneLake, or Google Cloud Storage; and modern data technologies such as Snowflake, Apache Iceberg, Apache Hudi, or BigQuery.
• Experience with relational, dimensional, logical, and physical data modeling; REST APIs and web services; data quality, metadata, governance, or lineage; and governed data products for analytics, AI, and Generative AI.
• Experience working with insurance and Health domain data.
• Strong experience building and supporting production-scale Databricks Lakehouse solutions and modern lakehouse architectures using Delta Lake.

Culture & Benefits

• Meritocratic culture that values self-starters and doing what is best for customers.
• Medical/Rx, dental, vision, life, and disability insurance.
• ESPP, 401(k), student loan assistance, and tuition reimbursement.
• Mental health and enhanced advocacy services.
• Paid time off, holidays, and preferred partner discounts; eligibility and waiting periods may vary.