Senior Director, Software Engineering (Data Engineering)

Location: United States; remote or hybrid to Chicago

Salary: $145,000–$190,000 annually, plus bonus eligibility.

Company

hirify.global supports continuous learning, feedback, diversity, equity, inclusion, and flexible work practices.

What you will do

• Design, build, and optimize scalable batch and real-time data pipelines using Python, PySpark, Apache Spark, and Airflow.
• Manage Azure-based cloud data infrastructure, processing costs, monitoring, automation, and CI/CD deployments.
• Apply data quality, governance, security, partitioning, caching, and performance-tuning practices across structured and unstructured data workloads.
• Design and maintain Airflow DAGs, including scheduling, dependencies, troubleshooting, and execution optimization.
• Lead and mentor data engineers while promoting coding, version-control, documentation, and Agile/Scrum practices.
• Collaborate with clients, engineering, product, architecture, and program management to deliver scalable products and align timelines and risks.

Requirements

• 8+ years of data engineering experience, including Python, PySpark, Spark, and Airflow.
• Strong Python and SQL programming skills with the ability to write efficient, maintainable code.
• Deep understanding of Spark internals, including RDDs, DataFrames, DAG execution, and partitioning.
• Experience with Airflow DAGs, scheduling, and dependency management.
• Knowledge of Git, Docker, Kubernetes, Terraform, and DevOps best practices for CI/CD workflows.
• Experience optimizing large-scale data processing, leading teams, and working in Agile/Scrum environments.
• Experience with common data warehouse and/or data lake platforms and knowledge of Azure, AWS, or Google Cloud technologies.
• Proven experience using generative AI tools for software development, data engineering, system design, testing, and troubleshooting, including prompt engineering, code generation, automated testing, and AI-driven productivity workflows.
• Ability to identify AI opportunities that improve team productivity, code quality, and operational efficiency, plus strong communication and collaboration skills for remote teams.

Nice to have

• Experience in ML engineering.

Culture & Benefits

• Flexible and accommodating environment supporting work and personal lives.
• Commitment to diversity, equity, inclusion, curiosity, accountability, client focus, and continuous improvement.
• Benefits may include paid time off, medical, dental, and vision insurance, and a 401(k) for eligible employees.
• The role is subject to AI-assisted resume screening.

Hiring process

• Applications may be submitted through the hirify.global careers website or the internal Intranet site.
• Prospective candidates may be asked to consent to background checks in accordance with local legislation and the candidate privacy notice.
• The position is expected to remain open for approximately 30 days and may close earlier once sufficient qualified candidates are identified.