Lead Data Engineer (ADB)

Location: Multiple locations; fully remote

Company

hirify.global connects job seekers with career opportunities, including projects for a Big Four professional services network developing enterprise technology and AI-driven tools for tax services.

What you will do

• Define and enforce coding standards and best practices across the project.
• Conduct code reviews and maintain high-quality, maintainable code.
• Design, develop, and maintain robust and scalable Spark applications.
• Optimize data processing code for performance, scalability, and reliability.
• Collaborate with cross-functional teams and communicate regularly with the client.
• Resolve technical issues and maintain documentation for code, processes, and workflows.

Requirements

• 5+ years of hands-on software development experience.
• Experience developing SQL stored procedures and migrating them to Spark SQL or PySpark.
• Extensive PySpark and Databricks experience, including Delta Tables, cluster management, and workflow automation.
• Production experience optimizing Spark performance and building data processing, analysis, and transformation workflows.
• Strong Python skills for complex data manipulation with Polars or Pandas, plus practical experience with Parquet and Delta Tables.
• Understanding of microservices architecture, strong analytical skills, and a pragmatic approach to technical problem-solving.

Nice to have

• Experience with Azure Cloud or other major cloud platforms, including Service Bus, Data Lake, Blob Storage, and Redis.
• Familiarity with FastAPI.
• Experience with Docker and Kubernetes.

Culture & Benefits

• Fully remote work across multiple locations.
• Collaboration with specialists in tax, software engineering, change management, and project management.
• Work on enterprise technology and AI-driven tools supporting tax services.