Senior Data Engineer (Databricks/PySpark)

TL;DR

Senior Data Engineer (Databricks/PySpark): Designing and operating scalable data pipelines to power BI and AI/ML products with an accent on Medallion Architecture and Data Fabric principles. Focus on building API-driven ingestion frameworks, optimizing PySpark workloads, and ensuring secure handling of sensitive government data.

Location: Vienna, VA, USA

Salary: $170,000 - $185,000 per year

Company

hirify.global is a government services provider delivering digital services, advanced analytics, and AI/ML solutions across various US federal agencies.

What you will do

• Design, develop, and optimize end-to-end data pipelines using Python and PySpark.
• Implement Medallion Architecture (Bronze, Silver, Gold) within a Databricks Lakehouse.
• Build data ingestion frameworks leveraging REST APIs, streaming APIs, and batch interfaces.
• Collaborate with architects on logical and physical data models to support BI&A and AI/ML products.
• Implement data validation, reconciliation, and secure handling of PII and sensitive data.
• Mentor junior and mid-level engineers on PySpark and API integration best practices.

Requirements

• Strong hands-on experience with Python, PySpark, and advanced SQL performance tuning.
• Proven experience building API-driven data ingestion pipelines (REST, JSON, OAuth).
• Deep expertise with Databricks, Delta Lake, and Lakehouse Architecture.
• Experience working with cloud platforms such as AWS, Azure, or GCP.
• Proven experience working in Agile Scrum environments.
• Must be based in or able to work in Vienna, VA, USA.

Nice to have

• Experience exposing data via internal or external APIs.
• Familiarity with streaming technologies like Kinesis, Kafka, or Event Hubs.
• Experience with data governance, cataloging, and lineage tools.
• Exposure to feature stores or ML data pipelines.

Culture & Benefits

• Comprehensive medical, dental, vision, life, STD, and LTD insurance.
• 401(k) plan with company matching.
• 11 federal holidays and paid time off (PTO).
• Tuition assistance and charitable contribution matching.
• Performance-based incentives for individual and team achievements.
• Flexible Spending Accounts for commuter, medical, and dependent care.