Lead Data Platform Engineer (Data Engineering)

TL;DR

Lead Data Platform Engineer (Data Engineering): Designing, building, and managing scalable data pipelines and systems, ensuring reliable, secure, and performant data flow across the organization. Focus on owning Data Lake and Databricks infrastructure, optimizing MapReduce workflows, and integrating real-time data streaming across complex distributed systems.

Company

hirify.global builds uncomplicated service software that delivers exceptional customer and employee experiences for over 72,000 companies globally.

What you will do

• Own and manage enterprise Data Lake infrastructure on AWS and Databricks.
• Design, develop, and optimize data ingestion and transformation pipelines from MySQL to Kafka and from Kafka to Databricks using Spark Structured Streaming.
• Design and implement MapReduce jobs to process and transform large-scale datasets.
• Build robust batch and near real-time data pipelines capable of handling high-volume, high-velocity data.
• Develop and maintain metadata-driven data processing frameworks.
• Work closely with Product, Regulatory, and Security teams to ensure compliance, privacy, and data quality.

Requirements

• 6-10 years of experience.
• Strong programming expertise in one or more of the following: Scala, Spark, Java, or Python.
• Proven experience working with Kafka for building event-driven or CDC-based pipelines.
• Hands-on experience with distributed data processing frameworks like Apache Spark, Databricks, or Flink.
• Solid understanding of Kubernetes for deploying and managing scalable, resilient data workloads (EKS preferred).
• Practical experience with AWS Cloud Services such as S3, Lambda, EMR, Glue, IAM, and CloudWatch.
• Experience designing and managing data lakehouse architectures using Databricks or similar platforms.
• Familiarity with metadata-driven frameworks and principles of data governance, lineage, and cataloging.
• Experience with CI/CD pipelines (Jenkins, GitHub Actions) and monitoring and alerting frameworks (Prometheus, Grafana, or ELK stack).

Culture & Benefits

• Environment that enables everyone to find their true potential, purpose, and passion.
• Welcoming colleagues of all backgrounds, genders, sexual orientations, religions, and ethnicities.
• Commitment to providing equal opportunity and fostering diversity in the workplace.
• Creating a vibrant, richer environment that boosts the goals of employees, communities, and business.