Director Of Engineering (AI)

TL;DR

Director Of Engineering (AI): Leading the end-to-end evaluation and experimentation lifecycle for AI agentic systems and traditional ML models, with an accent on defining technical vision, building critical infrastructure for automated evaluation systems, and designing online observability. Focus on integrating evaluation and experimentation pipelines into CI/CD workflows and driving adoption of best practices to ensure high-quality, safe, and performant AI systems.

Location: Onsite in New York, USA

Company

hirify.global is a leading AI Engineering organization building Agentforce, the future of AI-powered CRM.

What you will do

• Define and execute the technical vision for AI agent and ML model evaluation and experimentation.
• Own and build offline evaluation infrastructure, including regression testing and scenario-based simulations.
• Develop automated evaluation systems like LLM-as-Judge and hybrid scoring methods.
• Design and operate online evaluation, observability, and continuous performance monitoring.
• Lead the development of self-service evaluation and experimentation tooling for agent workflows.
• Lead and develop a senior engineering team, setting technical direction and fostering excellence.

Requirements

• Technical degree required.
• 10+ years of engineering experience, including 5+ years leading AI/ML teams.
• Proven ability to lead senior engineers and engineering managers.
• Strong experience building and operating experimentation platforms for AI systems.
• Deep understanding of LLM-based agentic architectures and traditional ML systems.
• Hands-on experience designing and integrating evaluation frameworks for AI systems into CI/CD pipelines.

Nice to have

• Master's or Ph.D. degree in computer science, machine learning, artificial intelligence, or related field.
• Experience with data and ML platforms (e.g., Snowflake-centric workflows, feature stores, training pipelines).
• Experience working in high-scale production AI/ML environments.

Culture & Benefits

• Wellbeing reimbursement.
• Generous parental leave and adoption assistance.
• Fertility benefits.
• Opportunity to gain deep experience building large-scale AI infrastructure.
• Shape the future of how hirify.global evaluates and ships AI systems.
• Make a direct impact on the quality and reliability of AI products used worldwide.