Назад
Company hidden
7 дней назад

AI Engineer (Evals)

Формат работы
onsite
Тип работы
fulltime
Грейд
lead
Английский
b2
Страна
UK
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
AI Engineer (Evals) (AI/Agent Evaluation): Building shared evaluation infrastructure, datasets, evaluators, benchmarks, and production feedback systems for trustworthy agents performing engineering and manufacturing workflows with an accent on deterministic checks, expert judgement, model-based grading, and human review. Focus on connecting offline evaluation, regression testing, telemetry, and cross-product scorecards while establishing rigorous evaluation practices across product, research, engineering, delivery, and customer teams.

Location: London, United Kingdom

Company

hirify.global is a physics AI company building simulation software and physics AI capabilities for industrial engineering and manufacturing across aerospace and defence, automotive, semiconductors, materials, and energy and renewables.

What you will do

  • Lead the technical strategy and implementation for evaluating AI agents.
  • Build reusable evaluation frameworks, reference implementations, golden paths, and production-quality tooling.
  • Help product, delivery, and customer teams establish robust evaluation suites and repeatable practices.
  • Develop datasets and evaluators combining deterministic checks, domain-expert judgement, model-based grading, and human review.
  • Connect offline evaluation, regression testing, and production telemetry to detect drift and learn from real usage.
  • Define cross-product scorecards, shared benchmarks, and public benchmarks for agents performing real engineering and manufacturing workflows.

Requirements

  • Experience building and operating evaluation systems for agents or LLM-powered products in production.
  • Ability to translate ambiguous product goals into datasets, evaluators, experiments, and trusted decision criteria.
  • Understanding of probabilistic-system failure modes and appropriate statistical judgement.
  • Knowledge of when to use deterministic evaluators, domain experts, human review, and LLM-as-a-judge, including their limitations.
  • Strong engineering skills for building usable frameworks and production-quality tooling.
  • Experience helping multiple teams adopt technical practices through delivery, standards, and coaching.
  • Ability to work with customers and domain experts to define and encode correctness in their workflows.
  • Knowledge of multi-step agent evaluation, tool calls, structured outputs, long-running workflows, trace-based observability, and OpenTelemetry concepts.
  • Knowledge of dataset design, sampling, experiment design, regression detection, reproducibility, CI, and release gates for non-deterministic systems.
  • Clear technical communication across product, research, engineering, delivery, and customer teams, with sound product judgement.

Nice to have

  • Experience with Pydantic Evals, Pydantic Logfire, or comparable code-first evaluation and observability tools.
  • Experience evaluating agents in engineering, simulation, design, manufacturing, or other expert domains.
  • Experience creating public benchmarks or open-source evaluation tooling.
  • Familiarity with post-training, synthetic data, reward signals, or model selection.
  • Experience in regulated, customer-operated, or air-gapped environments.

Culture & Benefits

  • Work across product, research, engineering, delivery, and customer teams.
  • Contribute to rigorous evaluation practices as an organisation-wide technical capability.
  • hirify.global is committed to equal employment opportunity and encourages applications from groups traditionally underrepresented in tech.
  • Diversity and inclusion data is collected confidentially and used only in aggregate to monitor equal-opportunity policies.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →