Назад
7 дней назад

Senior/Staff Evaluation Data Analyst (Agentic Search)

Формат работы
onsite
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
UK
Вакансия из списка Hirify.GlobalВакансия из Hirify RU Global, списка компаний с восточно-европейскими корнями
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Senior/Staff Evaluation Data Analyst (Agentic Search) (AI/search): Building evaluation systems, datasets, benchmarks, metrics, and automated pipelines to measure and improve search quality for AI agents with an accent on relevance, completeness, freshness, factual accuracy, and usefulness. Focus on investigating regressions and metric anomalies, applying statistical rigor to experimental design and measurement bias, and translating findings into product and engineering decisions.

Location: London, United Kingdom

Company

Nebius builds a full-stack AI cloud platform for data and model training through production deployment, including large-scale GPU orchestration, inference optimization, compute, storage, networking, and applied AI.

What you will do

  • Own the end-to-end quality measurement and improvement cycle, from identifying user needs and defining success to building evaluations, diagnosing failures, and improving outcomes with engineering and product teams.
  • Design, build, and evolve evaluation systems, including datasets, benchmarks, metrics, and automated pipelines supporting development and launch decisions.
  • Evaluate search response quality across relevance, completeness, freshness, factual accuracy, and usefulness to downstream AI agents.
  • Investigate quality regressions and metric anomalies, identify root causes, and prioritize high-impact improvements.
  • Support product and engineering roadmap decisions by making quality trade-offs explicit and assessing meaningful user impact.

Requirements

  • Substantial hands-on experience designing and building evaluation or quality measurement systems for complex technology products such as search, AI agents, voice technologies, autonomous driving, recommendation systems, or advertising.
  • Experience owning ambiguous problems from initial framing through implementation, diagnosis, and measurable product improvement.
  • Strong Python skills and an engineering mindset, including the ability to build reliable, maintainable pipelines and tools.
  • Solid foundation in mathematical statistics and practical experience with experimental design, sampling, uncertainty, and measurement bias.
  • Strong analytical judgment regarding what metrics capture, what they miss, and whether observed improvements matter to users.
  • Ability to reason across user needs, product behavior, and technical systems and translate that understanding into actionable evaluations.
  • Strong communication and collaboration skills, with the ability to influence technical and product decisions.
  • Independence and adaptability in a fast-moving environment where requirements, methods, and success definitions continue to evolve.
  • Applicants must be authorized to work in the country in which they apply and provide proof of employment eligibility as a condition of hire.

Culture & Benefits

  • Competitive compensation, career growth, and learning opportunities.
  • Flexibility, ownership, and a collaborative, innovative culture.
  • Opportunity to work on impactful AI projects with an international team.
  • Fast-moving environment emphasizing bold thinking, continuous growth, meaningful impact, and trust.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →