Lead Software Engineer (SparkFlow Framework)

TL;DR

Lead Software Engineer (SparkFlow Framework): Drive the evolution of the SparkFlow enterprise data processing framework built on Apache Spark with an accent on modularity, composability, and AI-driven developer tooling. Focus on designing composable abstractions, delivering new functional capabilities, and integrating the framework into the Unity control plane.

Location: Hybrid (3 days onsite in Charlotte, NC, 2 days remote)

Company

hirify.global is a staffing company offering contract opportunities with visa sponsorship and W2 employment.

What you will do

• Lead architectural evolution of the SparkFlow framework focusing on modularity and clear abstraction boundaries.
• Design and implement new functional features to expand framework capabilities.
• Ensure framework quality including performance, backward compatibility, and governance.
• Improve developer experience by simplifying APIs and onboarding patterns.
• Design and implement AI-driven solutions to enhance usability and productivity.
• Integrate SparkFlow into the Unity control plane and collaborate with cross-functional teams.

Requirements

• Location: Must be able to work hybrid with 3 days onsite in Charlotte, NC
• Deep experience with Apache Spark using Scala and/or Java; Python is a plus.
• Proven success building reusable frameworks or libraries with ownership of feature development.
• Strong skills in API design, abstraction design, and extensibility patterns.
• Experience applying AI/ML or LLM-based techniques to developer tooling or data frameworks.
• Experience working in large-scale enterprise data environments.

Nice to have

• Experience integrating frameworks into control planes or internal developer platforms.
• Familiarity with Spark SQL–centric architectures and data governance patterns.
• Experience modernizing or replacing legacy ETL frameworks.