Чёткая ответственность за важное направление оценки AI-поиска, сильные технические и статистические требования и влияние на продуктовые решения; быстрый темп подойдёт, если тебе комфортна неопределённость.
Высокий темп
Современный стек
Сфера AI
Чёткая зона влияния
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Подключить Plus
Описание вакансии
Текст:
Hirify AI/Оригинал
TL;DR
Senior/Staff Evaluation Data Analyst (Agentic Search) (AI/search): Building evaluation systems, datasets, benchmarks, metrics, and automated pipelines to measure and improve search quality for AI agents with an accent on relevance, completeness, freshness, factual accuracy, and usefulness. Focus on investigating regressions and metric anomalies, applying statistical rigor to experimental design and measurement bias, and translating findings into product and engineering decisions.
Location: London, United Kingdom
Company
Nebius builds a full-stack AI cloud platform for data and model training through production deployment, including large-scale GPU orchestration, inference optimization, compute, storage, networking, and applied AI.
What you will do
Own the end-to-end quality measurement and improvement cycle, from identifying user needs and defining success to building evaluations, diagnosing failures, and improving outcomes with engineering and product teams.
Design, build, and evolve evaluation systems, including datasets, benchmarks, metrics, and automated pipelines supporting development and launch decisions.
Evaluate search response quality across relevance, completeness, freshness, factual accuracy, and usefulness to downstream AI agents.
Investigate quality regressions and metric anomalies, identify root causes, and prioritize high-impact improvements.
Support product and engineering roadmap decisions by making quality trade-offs explicit and assessing meaningful user impact.
Requirements
Substantial hands-on experience designing and building evaluation or quality measurement systems for complex technology products such as search, AI agents, voice technologies, autonomous driving, recommendation systems, or advertising.
Experience owning ambiguous problems from initial framing through implementation, diagnosis, and measurable product improvement.
Strong Python skills and an engineering mindset, including the ability to build reliable, maintainable pipelines and tools.
Solid foundation in mathematical statistics and practical experience with experimental design, sampling, uncertainty, and measurement bias.
Strong analytical judgment regarding what metrics capture, what they miss, and whether observed improvements matter to users.
Ability to reason across user needs, product behavior, and technical systems and translate that understanding into actionable evaluations.
Strong communication and collaboration skills, with the ability to influence technical and product decisions.
Independence and adaptability in a fast-moving environment where requirements, methods, and success definitions continue to evolve.
Applicants must be authorized to work in the country in which they apply and provide proof of employment eligibility as a condition of hire.
Culture & Benefits
Competitive compensation, career growth, and learning opportunities.
Flexibility, ownership, and a collaborative, innovative culture.
Opportunity to work on impactful AI projects with an international team.
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →