Director, Research - AI Evals (AI)

TL;DR

Director, Research - AI Evals (AI): Leading the strategy and execution of quality measurement for AI-powered product experiences with an accent on defining quality dimensions, building evaluation frameworks, and establishing rigorous testing standards. Focus on bridging human evaluation with automated model-based approaches to provide actionable signals for product shipping decisions.

Location: Must be based in the United States (Remote or hybrid from San Francisco/New York hubs).

Salary: $258,000–$348,000 USD

Company

Figma is a collaborative design platform empowering teams to brainstorm, create, and iterate on products in real time from anywhere in the world.

What you will do

• Define quality dimensions and evaluation methods for AI-powered features.
• Build and maintain evaluation frameworks, rubrics, and golden datasets.
• Partner with engineering to implement repeatable evaluation pipelines and regression testing.
• Produce clear dashboards and readouts to guide product prioritization and shipping decisions.
• Advocate for evaluation standards across cross-disciplinary teams.
• Manage a team to execute AI evaluations using internal staff, contractors, and LLMs.

Requirements

• 10+ years of experience in product, research, or applied research.
• 2+ years of management experience.
• Direct, hands-on experience evaluating AI/LLM-powered products.
• Expertise in human evaluation programs, rubric construction, and inter-rater reliability.
• Strong ability to reason about model behavior and apply automated/model-based evaluation approaches.
• Proven track record of gaining buy-in from executive and cross-disciplinary stakeholders.

Nice to have

• Experience with eval tooling such as Braintrust, LangSmith, or DeepEval.
• Experience standing up a new function or discipline from scratch.
• Background in product design, data science, or front-end engineering.
• Familiarity with Figma products.

Culture & Benefits

• Comprehensive health, dental, and vision insurance.
• Retirement plan with company contribution.
• Generous PTO, company recharge days, and parental leave.
• Learning & development stipend and work-from-home support.
• Annual bonus plan for eligible roles.