Lead Databricks Platform Engineer (Remote US)

Location: Remote within the United States; candidates must reside in the U.S., be authorized to work in the U.S., and perform all work in the U.S.

Salary: $98,614.00–$202,850.00 annually

Company

hirify.global is a global advisory and technology services provider combining consulting expertise with technology to solve complex client challenges.

What you will do

• Design, configure, secure, and operate Azure Databricks workspaces, clusters, jobs, workflows, catalogs, storage credentials, and service principals.
• Build and optimize batch and real-time ingestion and transformation pipelines using Python, PySpark, SQL, Delta Lake, Delta Live Tables, Auto Loader, and Apache Spark.
• Implement medallion architecture, dimensional models, change data capture, schema evolution, data quality controls, and reliable processing patterns.
• Configure Unity Catalog governance and secure integrations with ADLS Gen2 and other enterprise data services.
• Automate deployments and environment configuration with Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable tools.
• Monitor platform health, troubleshoot performance and data issues, and support security, compliance, business continuity, disaster recovery, and operational readiness.

Requirements

• More than 5 years of data engineering experience, including at least 3 years supporting Azure Databricks and related Azure data services.
• Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala, with experience designing and optimizing large-scale ETL/ELT pipelines.
• Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Databricks workflows, data modeling, governance, security, and performance troubleshooting.
• Experience implementing CI/CD or infrastructure as code for data platforms using Git and tools such as Terraform, Databricks Asset Bundles, GitHub Actions, or Azure DevOps.
• U.S. citizenship, U.S. residence, U.S. work authorization, and the ability to satisfy FDIC personnel eligibility and background-investigation requirements are required.
• Ability to communicate architecture, risks, trade-offs, and operational impacts to technical and nontechnical stakeholders.

Nice to have

• Databricks, Unity Catalog, Delta Live Tables, Auto Loader, Kafka, Confluent, Structured Streaming, Snowflake, Apache Iceberg, or other enterprise data platform experience.
• Experience with generative AI, NLP, RAG applications, vector stores, agents, MLflow, or Databricks AI/ML workloads.
• Experience with cross-cloud data and AI architectures, Docker, Kubernetes, REST APIs, FastAPI, or Node.js.
• Experience supporting ATO processes, NIST or federal compliance controls, vulnerability remediation, Zero Trust, BCDR, and disaster recovery.
• Databricks Data Engineer certification or comparable demonstrated experience.

Culture & Benefits

• Full-time remote work within the United States.
• Collaboration across data engineering, AI, cloud, security, application, operations, and agency stakeholder groups.
• Participation in Agile planning, code reviews, demos, and retrospectives.
• Work includes architecture documentation, runbooks, data models, governance decisions, and deployment procedures in version-controlled repositories.
• Reasonable accommodations are available throughout the application and employment process.