Lead Data Platform and ML Engineer (Databricks/AWS)

Location: Remote anywhere in the United States, with periodic travel. Offices are in Conshohocken, Pennsylvania, near Philadelphia.

Company

Lighthouse Life is part of LHL Strategies, an integrated platform serving asset managers that acquire and manage life insurance policies and portfolios across acquisition, trading, valuation, management, and servicing.

What you will do

• Operate and improve production data pipelines bringing structured, unstructured, operational, and document data into a Databricks lakehouse.
• Extend the lakehouse, curated dbt data products, semantic layer, and Databricks AI/BI dashboards with quality checks and governance built in.
• Enforce data ownership, sharing, ingestion, access, lineage, sensitive-data handling, and auditability requirements through Unity Catalog.
• Own infrastructure and workloads as code, automated CI/CD, data quality checks, freshness monitoring, anomaly detection, cloud costs, runbooks, and documentation.
• Build and operate document extraction with Databricks AI functions and Amazon Textract, and develop, validate, govern, deploy, and monitor machine learning models.
• Run the MLflow lifecycle and batch and real-time Databricks Model Serving, including retraining, evaluation, drift response, explainability, and fairness testing.

Requirements

• At least 5 years of experience building production data pipelines, with strong SQL and Python, production Spark or PySpark, and dbt transformation and testing experience.
• Hands-on production experience with Databricks, including Unity Catalog, Delta Lake, Lakeflow, Databricks SQL, and Databricks AI functions.
• Experience running Databricks on AWS and using Terraform or OpenTofu for infrastructure as code and Databricks Asset Bundles for workloads.
• Experience owning GitHub Actions CI/CD pipelines with automated testing and controlled promotion across environments.
• Experience building and shipping machine learning models end to end, including feature pipelines, experiment tracking, MLflow, and batch and real-time serving.
• Experience with drift detection and response, retraining strategy, champion/challenger evaluation, and explainability methods such as SHAP.
• Experience handling regulated data, including HIPAA or equivalent, PHI and PII protection, encryption, least-privilege access, and audit logging.
• Regular use of AI-assisted development tools such as Claude Code, Databricks Assistant, or GitHub Copilot, with careful review of generated output.
• Bachelor’s degree in Computer Science, Statistics, Mathematics, Engineering, or a related quantitative field, or equivalent practical experience. Excellent written and verbal communication skills are required.

Nice to have

• Experience with document extraction at scale.
• Experience taking over and hardening a platform built by another team.
• Master’s degree or experience in insurance, actuarial science, healthcare, or credit risk modeling.

Culture & Benefits

• Remote-first, distributed work with high transparency, daily standups, asynchronous communication, and ownership of outcomes.
• Collaborative, entrepreneurial environment with experienced leadership and close work with development, business, analytics, and compliance teams.
• Comprehensive health, dental, and vision coverage, 401(k), and flexible PTO.
• Work focused on finance, insurance, and machine learning in an underserved market.