Agentic AI Optimization Developer (AI)

Location: Toronto, Canada

Base salary: CAD 78,504–103,037 per year

Company

hirify.global is a data, analytics, and technology company supporting financial institutions, employers, government agencies, and consumers with information-driven decision-making.

What you will do

• Curate and maintain Golden Sets containing documents, user queries, and expected agent trajectories.
• Design continuous automated evaluation pipelines for agent accuracy, latency, token spend, and fallback reliability.
• Audit multi-step agent trajectories and reasoning loops to identify deviations from intended logic.
• Optimize system prompts, context windows, few-shot examples, function calling, external API integrations, databases, and UiPath workflows.
• Monitor production agents for semantic drift, prompt injection, infinite loops, hallucinations, and execution failures.
• Improve RAG pipelines, knowledge bases, and guardrails in collaboration with AI Solution Leads and AI Agent Developers.

Requirements

• 3+ years of experience in software quality engineering, test automation, or data/ML engineering.
• Recent experience with LLM testing, prompt tuning, or orchestration patterns.
• Hands-on experience with LLM orchestration frameworks such as LangGraph, ADKs, or specialized SDKs.
• Strong understanding of JSON schema design, LLM tool calling, function calling, and structured outputs.
• Strong Python-based test automation and experience debugging asynchronous, non-deterministic systems using tracing and observability.

Nice to have

• Experience with AI evaluation and observability platforms or live production testing of agentic workflows.
• Familiarity with Google Cloud Vertex AI, Gemini Enterprise Agent Platform, or UiPath Maestro.
• Experience generating secure synthetic data for edge-case testing with LLMs.
• Proficiency in TypeScript, asynchronous programming, API design, or microservices architecture.

Culture & Benefits

• Comprehensive compensation and healthcare packages.
• Paid time off.
• Online learning platform with guided career tracks and organizational growth opportunities.
• Inclusive, equitable, and accessible workplace with accommodations available during recruitment.