Назад
Company hidden
обновлено 2 дня назад

AI Agent Evaluation Engineer

Формат работы
remote (только USA)
Тип работы
fulltime
Английский
b2
Страна
US/Canada
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
AI Agent Evaluation Engineer (LLM Evaluation/Conversational AI): Building and operating evaluation systems for production AI agents across conversations, with an accent on rubric design, automated judges, human calibration, and multilingual quality analysis. Focus on identifying failure modes, constructing golden sets and regression suites, sampling conversations, and defining reliable quality and latency thresholds.

Location: Remote in Canada or the United States; listed Canadian locations include Alberta, British Columbia, and Ontario.

Company

hirify.global develops a unified AI platform combining conversational AI agents, real-time human agent augmentation, and conversation intelligence for customer experience operations.

What you will do

  • Define and maintain evaluation rubrics covering adherence, outcomes, and conversation experience, including latency, turn-taking, tone, repetition, tool use, and guardrails.
  • Write annotation guidelines and failure taxonomies, run calibration sessions, measure inter-rater agreement, and prioritize failures by severity and frequency.
  • Review and annotate production conversations, identify likely causes, and route evidence-based findings to engineering and product owners.
  • Build and operate a pipeline combining automated checks, LLM-based judges, human review, sampling strategies, golden sets, and regression suites.
  • Design coverage for multilingual behavior and high-stakes edge cases such as human handoffs, unwanted callers, misuse, and upstream failures.
  • Report quality trends and methodology to stakeholders and customers, and collaborate with engineering, product, and voice/language teams on prioritized recommendations.

Requirements

  • 3+ years of professional experience in software engineering, applied ML, or data-intensive research, including at least 1–2 years working hands-on with LLM-based systems in production.
  • Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, a related field, or equivalent practical experience.
  • Experience building and operating an end-to-end evaluation system for a production LLM product, including a rubric, human labels, an automated judge, and a regression suite.
  • Deep understanding of evaluation methodology, including golden sets, judges, regression suites, calibration, and their failure modes; tooling experience is tool-agnostic.
  • Understanding of LLM agents, including prompts, tool calls, context handling, and multi-turn state, with the ability to interpret agent traces.
  • Strong Python programming skills and experience with LLM APIs; able to build and ship evaluation tooling independently. Fine-tuning and ML frameworks are not required.
  • Ability to create consistently scored rubrics, measure reviewer agreement, and design tests from open-ended quality questions.
  • Willingness to perform weekly hands-on conversation listening, labeling, and failure-taxonomy work, independently and with QA partners.
  • Professional fluency in at least one language other than English, sufficient to evaluate multilingual agent behavior directly. Strong written communication skills are required.

Culture & Benefits

  • Remote work with a setup budget for a productive home office.
  • Medical, dental, and vision coverage for employees and families.
  • Flexible PTO and paid parental leave for all new parents.
  • Retirement savings plan, monthly wellness and communication stipend, and onsite meal and commuter benefits for onsite employees.
  • Location-based base pay, with final offers shaped by experience, skills, education, and geography; equity and benefits are included in total compensation.

Hiring process

  • After the initial resume review, qualified applicants complete a self-paced CoderPad technical assessment before conversational interviews.
  • Applicants may opt out of AI tools used in recruiting by contacting recruiting@hirify.global.ai with the subject “Opt-Out”. Candidates may not record or use AI transcription during interviews without an approved exception or accommodation.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →