Чёткая исследовательская роль в оценке передовых ИИ-систем сочетает практическую работу, вклад в открытые исследования и возможность влиять на качество бенчмарков.
На передовой ИИ
Понятная зона задач
Влияние на исследования
Современный стек
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Подключить Plus
Описание вакансии
Текст:
Hirify AI/Оригинал
TL;DR
Research Scientist, Open Benchmarks Red Team (AI Evaluation): Identifying weaknesses in frontier AI benchmarks and developing, implementing, and validating repairs with an accent on adversarial testing, AI evaluation, and benchmark reliability. Focus on reproducing grader failures, reward hacking, task ambiguity, shortcuts, and environment vulnerabilities while building reusable methods for evaluation across tasks and datasets.
Location: Hybrid in New York City, NY or San Francisco, CA; remote within the United States
Salary: $198,000–$340,000 USD annually
Company
Snorkel AI is an AI data lab developing datasets, benchmarks, evaluations, and custom solutions for frontier and agentic AI systems.
What you will do
Red-team frontier AI benchmarks by identifying and reproducing reward hacking, grader failures, task ambiguity, shortcuts, and environment vulnerabilities.
Propose, implement, and validate benchmark repairs with benchmark authors without breaking valid solutions.
Reuse existing tools and develop reusable checks for recurring weaknesses across tasks and datasets.
Review findings with benchmark authors, collaborate on repairs, and support improved benchmark releases.
Contribute to technical reports, open-source tools, publications, and new approaches to benchmark evaluation.
Requirements
Strong research or engineering background in machine learning, LLMs, AI agents, and evaluation.
Strong Python skills and experience building, running, and debugging AI agents or evaluation systems.
Ability to identify weaknesses in evaluation methods, develop new approaches, and rigorously test their effectiveness.
Experience designing rigorous experiments, analyzing model behavior, and investigating unexpected failures.
Familiarity with LLM or agent benchmarks, evaluation frameworks, or automated grading systems.
Experience independently driving research and engineering projects from problem identification through implementation and validation.
Strong communication and collaboration skills.
Nice to have
Experience with adversarial testing, red teaming, reward hacking, or specification gaming.
Experience developing or evaluating benchmarks for coding agents, computer use, or professional workflows.
Familiarity with reinforcement learning environments, agent trajectories, verifiers, and evaluation infrastructure.
Open-source contributions or publications in AI evaluation, robustness, or agentic systems.
PhD in Computer Science, Machine Learning, or a related field, or equivalent research or industry experience.
Culture & Benefits
Work in a research-driven AI company focused on high-performing frontier and agentic AI.
Meaningful opportunities to shape technical priorities, influence strategic decisions, and contribute to open research and tools.
Environment focused on career growth, learning, technical expertise, and cross-functional development.
Equal employment opportunity employer committed to a diverse and inclusive workplace.
Будьте осторожны
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →