Agentic QA Engineer (AI)

Location: Toronto, Ontario, Canada

Company

hirify.global is a digital innovation firm that partners with clients to build digital products, modernize technology platforms, and apply AI to business challenges.

What you will do

• Design testing strategies for AI agents, LLM-powered applications, and agentic workflows.
• Build automated functional, integration, regression, and end-to-end test frameworks.
• Develop evaluation pipelines, datasets, scenarios, metrics, and benchmarks for accuracy, reliability, task completion, and consistency.
• Test agent tool usage, API integrations, multi-step reasoning, and external-system interactions.
• Identify, reproduce, and troubleshoot hallucinations, incorrect tool calls, unexpected outputs, workflow breakdowns, and production issues.
• Integrate quality checks into CI/CD and collaborate with AI engineers, developers, product teams, and client-facing engineering teams.

Requirements

• Professional experience in Software QA, Quality Engineering, SDET, Test Automation, or Software Engineering with a strong testing focus.
• Proficiency in Python, JavaScript, TypeScript, or another relevant programming language.
• Experience building automated tests and testing frameworks.
• Strong understanding of API, integration, regression, and end-to-end testing.
• Experience debugging applications and investigating complex technical issues.
• Familiarity with Git, CI/CD pipelines, and modern software development practices.
• Strong analytical thinking and the ability to design test cases for complex, real-world scenarios.

Nice to have

• Experience testing LLM-based applications, AI agents, or generative AI products.
• Familiarity with LangGraph, LangChain, CrewAI, or similar agentic frameworks.
• Experience with LLM evaluation, prompt testing, model benchmarking, or AI observability.
• Familiarity with Pytest, Playwright, Cypress, or Selenium.
• Understanding of hallucinations, nondeterministic outputs, prompt injection, and agent tool-use failures.
• Experience with cloud platforms, APIs, distributed systems, and production deployments.
• Previous experience in Agentic Engineering, Forward Deployed Engineering, Applied AI Engineering, or AI product development.

Culture & Benefits

• Collaborative client-partnership model spanning strategy through execution.
• Work focused on AI-driven innovation, practical innovation, ownership, transparency, and empowerment.
• Equal employment opportunity and accommodations for applicants with disabilities.