Назад
Company hidden
16 минут назад

Senior AI Engineer - Agentic AI Evaluation

Формат работы
hybrid
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
Singapore/Germany
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Senior AI Engineer - Agentic AI Evaluation (Agentic AI/Python): Building the evaluation engine for tool-using, multi-step agents with an accent on stateful behavior, evaluator integrity, security, and production reliability. Focus on designing trustworthy benchmarks, testing adversarial agent behavior, translating AI governance requirements into evaluation logic, and extending assessment to autonomous systems.

Location: Hybrid in Munich, Germany or Singapore

Company

hirify.global is an independent third-party AI assurance company building software and evaluations for trustworthy deployment of computer vision, LLM, vision-language, agentic, and autonomous AI systems in government, defence, and regulated commercial sectors.

What you will do

  • Own and build the agentic-evaluation engine for scoring intent resolution, tool selection and parameterisation, task adherence, planning, tool-call ordering, response completeness, and human handoff.
  • Design harnesses for stateful, multi-turn agents to expose behavioral drift, degradation, and safety issues across long sessions.
  • Build trustworthy evaluations by isolating runners, checking semantics, treating crashed or incomplete tasks as failures, auditing ground truth, and testing the evaluation setup itself.
  • Partner on red-teaming and adversarial evaluation covering tool-chain exploits, RBAC and permission-inheritance failures, and multi-turn safety erosion.
  • Translate EU AI Act, NIST AI RMF, and ISO 42001 requirements into testable evaluation logic and close the lab-to-production assurance gap.
  • Set the technical direction through spikes, ADRs, code review, and future extensions for embodied and autonomous systems.

Requirements

  • 5+ years of professional software engineering experience, including shipping and operating production systems in Python; prototype-only experience is insufficient.
  • Demonstrated depth in agentic AI, including building, orchestrating, or rigorously evaluating LLM agents and understanding tool calling, memory, multi-step control flow, MCP, and A2A beyond tutorial level.
  • Knowledge of execution-based benchmarks including GAIA, τ²-bench, OSWorld, SWE-bench Verified, and WebArena, including their limitations and failure modes.
  • Strong engineering judgment with the ability to scope ambiguous research-adjacent problems and drive them to shipped, tested, observable outcomes independently.
  • Hands-on technical depth combined with clear documentation, communication, and effective handover.
  • Clear written and verbal communication and comfort working toward concrete quarterly outcomes across a distributed Singapore-Europe team.

Nice to have

  • Agentic security or red-teaming experience, including adversarial testing, jailbreaks, tool-chain exploits, or agent identity and on-behalf-of authorisation.
  • AI evaluation, testing, or assurance tooling experience, including LLM/RAG, computer vision, LLM-as-judge pipelines, and their failure modes.
  • Familiarity with AI governance frameworks and translating principles into engineering requirements.
  • Exposure to embodied or autonomous AI, including vision-language-action models, simulators, or world models.
  • Experience with data-intensive pipelines using Parquet, Iceberg, DuckDB, or similar technologies.
  • Experience with container-based or serverless evaluation frameworks, Kubernetes/Helm, or GPU/CUDA infrastructure.
  • Experience in regulated, safety-critical, or defence contexts where evidence, auditability, and reproducibility were first-class requirements.

Culture & Benefits

  • Hands-on individual-contributor role with domain ownership and no line management responsibility.
  • Close partnership with Product, governance analysts, and an Engineering Manager responsible for the evaluation engine.
  • High, transparent engineering bar focused on production-grade, tested, observable, maintainable code.
  • Distributed collaboration across Singapore and Europe with emphasis on clear communication, documented decisions, independent operation, and effective handover.
  • Equal opportunity workplace supporting diverse cultures, perspectives, skills, and experiences.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →