Вакансия из Hirify Global, списка международных tech-компаний Для мэтча и отклика нужен Plus
Пожаловаться
Зарплата и рынок
ЗП не указанаразвернуть
На похожих ролях по рынку
$122к$157кмедиана$182к
23 в выборке
На СНГ рынке: $4.3к/мес ($3к - $5.7к)
91
Крутая вакансия
развернуть
Чётко очерченная практическая lead-роль в оценке production-агентов с влиянием на работу разных команд и связью с промышленным ИИ; уровень оплаты и повседневные условия работы не раскрыты.
Понятная зона ответственности
ИИ для промышленности
Влияние на всю организацию
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Подключить Plus
Описание вакансии
Текст:
Hirify AI/Оригинал
TL;DR
AI Engineer (Evals) (AI/Agent Evaluation): Building shared evaluation infrastructure, datasets, evaluators, benchmarks, and production feedback systems for trustworthy agents performing engineering and manufacturing workflows with an accent on deterministic checks, expert judgement, model-based grading, and human review. Focus on connecting offline evaluation, regression testing, telemetry, and cross-product scorecards while establishing rigorous evaluation practices across product, research, engineering, delivery, and customer teams.
Location: London, United Kingdom
Company
hirify.global is a physics AI company building simulation software and physics AI capabilities for industrial engineering and manufacturing across aerospace and defence, automotive, semiconductors, materials, and energy and renewables.
What you will do
Lead the technical strategy and implementation for evaluating AI agents.
Build reusable evaluation frameworks, reference implementations, golden paths, and production-quality tooling.
Help product, delivery, and customer teams establish robust evaluation suites and repeatable practices.
Develop datasets and evaluators combining deterministic checks, domain-expert judgement, model-based grading, and human review.
Connect offline evaluation, regression testing, and production telemetry to detect drift and learn from real usage.
Define cross-product scorecards, shared benchmarks, and public benchmarks for agents performing real engineering and manufacturing workflows.
Requirements
Experience building and operating evaluation systems for agents or LLM-powered products in production.
Ability to translate ambiguous product goals into datasets, evaluators, experiments, and trusted decision criteria.
Understanding of probabilistic-system failure modes and appropriate statistical judgement.
Knowledge of when to use deterministic evaluators, domain experts, human review, and LLM-as-a-judge, including their limitations.
Strong engineering skills for building usable frameworks and production-quality tooling.
Experience helping multiple teams adopt technical practices through delivery, standards, and coaching.
Ability to work with customers and domain experts to define and encode correctness in their workflows.
Knowledge of multi-step agent evaluation, tool calls, structured outputs, long-running workflows, trace-based observability, and OpenTelemetry concepts.
Knowledge of dataset design, sampling, experiment design, regression detection, reproducibility, CI, and release gates for non-deterministic systems.
Clear technical communication across product, research, engineering, delivery, and customer teams, with sound product judgement.
Nice to have
Experience with Pydantic Evals, Pydantic Logfire, or comparable code-first evaluation and observability tools.
Experience evaluating agents in engineering, simulation, design, manufacturing, or other expert domains.
Experience creating public benchmarks or open-source evaluation tooling.
Familiarity with post-training, synthetic data, reward signals, or model selection.
Experience in regulated, customer-operated, or air-gapped environments.
Culture & Benefits
Work across product, research, engineering, delivery, and customer teams.
Contribute to rigorous evaluation practices as an organisation-wide technical capability.
hirify.global is committed to equal employment opportunity and encourages applications from groups traditionally underrepresented in tech.
Diversity and inclusion data is collected confidentially and used only in aggregate to monitor equal-opportunity policies.
Будьте осторожны
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →