AI Evaluation Engineer (AI)

TL;DR

AI Evaluation Engineer (AI): Designing and executing validation strategies for agentic, NLP, and speech workflows with an accent on LLM-judge metric development and benchmark curation. Focus on building scalable evaluation pipelines, performing red teaming analyses, and ensuring release-readiness for complex agentic voice and chat solutions.

Location: Must be based in or able to work from Kitchener or Vancouver, Canada

Salary: $104,000–$119,750 CAD

Company

hirify.global is an AI-powered customer experience platform that resolves customer issues in real-time through advanced agentic AI and human-in-the-loop collaboration.

What you will do

• Design and execute validation strategies for agentic, NLP, and speech workflows across staging and production environments.
• Develop and calibrate LLM-judge metrics and refine prompts to improve evaluation accuracy.
• Build and maintain QA tooling, notebooks, and automated pipelines for scalable regression testing.
• Configure and monitor data annotation jobs to maintain high-quality evaluation datasets.
• Perform red teaming and A/B comparisons to validate model changes before release.
• Investigate bugs and triage issues in collaboration with applied science and engineering teams.

Requirements

• Must be based in Canada with the ability to work from Kitchener or Vancouver offices.
• Bachelor's or Master's degree in Computer Science, Software Engineering, or Computational Linguistics.
• 3+ years of experience in QA, test engineering, or model evaluation for AI-driven products.
• Strong experience designing structured test strategies for both manual and automated workflows.
• Proficiency in working with complex AI systems including LLMs, NLP, or speech technologies.
• Strong analytical skills for investigating failures and identifying actionable quality patterns.

Culture & Benefits

• Work at the forefront of AI transformation in business communications.
• Comprehensive benefits package and competitive salary.
• Inclusive office environment designed to foster collaboration and connection.
• Robust training programs and opportunities for professional growth.
• Access to cutting-edge AI tools to enhance daily productivity.