Назад
7 дней назад

Research Scientist, Open Benchmarks Red Team (AI Evaluation)

198 000 - 340 000$
Формат работы
remote (только USA)/hybrid
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Research Scientist, Open Benchmarks Red Team (AI Evaluation): Identifying weaknesses in frontier AI benchmarks and developing, implementing, and validating repairs with an accent on adversarial testing, AI evaluation, and benchmark reliability. Focus on reproducing grader failures, reward hacking, task ambiguity, shortcuts, and environment vulnerabilities while building reusable methods for evaluation across tasks and datasets.

Location: Hybrid in New York City, NY or San Francisco, CA; remote within the United States

Salary: $198,000–$340,000 USD annually

Company

Snorkel AI is an AI data lab developing datasets, benchmarks, evaluations, and custom solutions for frontier and agentic AI systems.

What you will do

  • Red-team frontier AI benchmarks by identifying and reproducing reward hacking, grader failures, task ambiguity, shortcuts, and environment vulnerabilities.
  • Propose, implement, and validate benchmark repairs with benchmark authors without breaking valid solutions.
  • Reuse existing tools and develop reusable checks for recurring weaknesses across tasks and datasets.
  • Review findings with benchmark authors, collaborate on repairs, and support improved benchmark releases.
  • Contribute to technical reports, open-source tools, publications, and new approaches to benchmark evaluation.

Requirements

  • Strong research or engineering background in machine learning, LLMs, AI agents, and evaluation.
  • Strong Python skills and experience building, running, and debugging AI agents or evaluation systems.
  • Ability to identify weaknesses in evaluation methods, develop new approaches, and rigorously test their effectiveness.
  • Experience designing rigorous experiments, analyzing model behavior, and investigating unexpected failures.
  • Familiarity with LLM or agent benchmarks, evaluation frameworks, or automated grading systems.
  • Experience independently driving research and engineering projects from problem identification through implementation and validation.
  • Strong communication and collaboration skills.

Nice to have

  • Experience with adversarial testing, red teaming, reward hacking, or specification gaming.
  • Experience developing or evaluating benchmarks for coding agents, computer use, or professional workflows.
  • Familiarity with reinforcement learning environments, agent trajectories, verifiers, and evaluation infrastructure.
  • Open-source contributions or publications in AI evaluation, robustness, or agentic systems.
  • PhD in Computer Science, Machine Learning, or a related field, or equivalent research or industry experience.

Culture & Benefits

  • Work in a research-driven AI company focused on high-performing frontier and agentic AI.
  • Meaningful opportunities to shape technical priorities, influence strategic decisions, and contribute to open research and tools.
  • Environment focused on career growth, learning, technical expertise, and cross-functional development.
  • Equal employment opportunity employer committed to a diverse and inclusive workplace.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →