Назад
7 дней назад

Researcher, AI Evaluations (AI)

169 000 - 250 000$
Формат работы
remote (только USA)/onsite
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Researcher, AI Evaluations (AI): Designing and running evaluation research for AI models, agent systems, and products with an accent on human evaluations, quality rubrics, representative datasets, benchmarks, and automated graders. Focus on validating LLM-as-judge methods, analyzing uncertainty and failure patterns, and integrating reliable evaluation methods into regression testing and model improvement workflows.

Location: San Francisco, CA; New York, NY; or remotely in the United States. The role can also be held from one of Figma's US hubs. In-person onboarding is required if hired.

Annual base salary: $169,000–$250,000 USD.

Company

Figma develops a collaborative design platform for brainstorming, prototyping, translating designs into code, and iterating with AI.

What you will do

  • Own evaluation research for priority AI Research workstreams by defining quality criteria, designing studies, analyzing results, and recommending actions.
  • Build rubrics, representative evaluation datasets, benchmarks, and quality thresholds grounded in user needs and expert design judgment.
  • Partner with AI Research and Modeling Platform engineers to integrate validated methods into repeatable evaluations and regression tests.
  • Explain quality, failure patterns, and uncertainty to guide model selection, model improvements, product priorities, and release decisions.
  • Validate automated graders against human ratings, including LLM-as-judge methods, and identify bias, gaps, and uncertainty.
  • Connect user research with model development, share reusable methods, review work, and mentor peers.

Requirements

  • 7+ years of experience in research, applied research, or a related field, including independently leading complex evaluation projects and delivering measurable impact.
  • Hands-on experience evaluating AI products and translating evaluation findings into model and product improvements.
  • Expertise designing human evaluations, scoring rubrics, datasets, and benchmarks, including measuring inter-rater agreement and validating automated graders.
  • Strong qualitative and quantitative research skills, including data analysis, uncertainty interpretation, and actionable insights into model behavior and performance.
  • Excellent problem-solving and stakeholder management skills, with the ability to navigate ambiguity, prioritize risks, balance speed with rigor, and influence cross-functional decisions.

Nice to have

  • Experience integrating automated evaluations and regression testing into engineering workflows, using tools such as Braintrust, LangSmith, DeepEval, or equivalent platforms.
  • Experience developing scalable, reusable evaluation frameworks and driving consistent quality standards across teams.
  • Background in product design, product management, data science, or engineering, with familiarity with Figma or similar design and developer tools.
  • Strong statistical expertise in sampling, significance testing, and agreement metrics, including experiments or performance-metric analysis for AI-powered features.
  • Advanced training in a quantitative or behavioral discipline through formal education or equivalent hands-on experience.

Culture & Benefits

  • Full-time senior individual contributor role with no direct reports, reporting to the AI Insights lead and working with AI Research within AI Platform.
  • Employees receive equity and benefits including health, dental, vision, retirement contributions, paid parental and family-planning leave, mental-health support, paid time off, sick leave, holidays, and flexible PTO for eligible exempt employees.
  • Additional benefits may include recharge days, cell phone and home internet reimbursements, and lifestyle spending accounts.
  • Video interviews require candidates to keep cameras on; in-person onboarding is required after hiring.
  • Figma is an equal opportunity workplace and provides reasonable accommodations throughout the application and employment process.

Hiring process

  • Video interviews require the candidate's camera to remain on.
  • Candidates may submit up to 5 applications during any rolling 30-day period.
  • Candidates may not reapply to the same role while an application is active or within 90 days after rejection.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →