В Figma чётко очерчена senior-роль в оценке AI с влиянием на решения по продукту и моделям, а сильная продуктовая среда добавляет привлекательности; ограничение по локации сужает доступ.
Только США
Оценка AI
Самостоятельная роль
Сильный продукт
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Подключить Plus
Описание вакансии
Текст:
Hirify AI/Оригинал
TL;DR
Researcher, AI Evaluations (AI): Designing and running evaluation research for AI models, agent systems, and products with an accent on human evaluations, quality rubrics, representative datasets, benchmarks, and automated graders. Focus on validating LLM-as-judge methods, analyzing uncertainty and failure patterns, and integrating reliable evaluation methods into regression testing and model improvement workflows.
Location: San Francisco, CA; New York, NY; or remotely in the United States. The role can also be held from one of Figma's US hubs. In-person onboarding is required if hired.
Annual base salary: $169,000–$250,000 USD.
Company
Figma develops a collaborative design platform for brainstorming, prototyping, translating designs into code, and iterating with AI.
What you will do
Own evaluation research for priority AI Research workstreams by defining quality criteria, designing studies, analyzing results, and recommending actions.
Build rubrics, representative evaluation datasets, benchmarks, and quality thresholds grounded in user needs and expert design judgment.
Partner with AI Research and Modeling Platform engineers to integrate validated methods into repeatable evaluations and regression tests.
Explain quality, failure patterns, and uncertainty to guide model selection, model improvements, product priorities, and release decisions.
Validate automated graders against human ratings, including LLM-as-judge methods, and identify bias, gaps, and uncertainty.
Connect user research with model development, share reusable methods, review work, and mentor peers.
Requirements
7+ years of experience in research, applied research, or a related field, including independently leading complex evaluation projects and delivering measurable impact.
Hands-on experience evaluating AI products and translating evaluation findings into model and product improvements.
Expertise designing human evaluations, scoring rubrics, datasets, and benchmarks, including measuring inter-rater agreement and validating automated graders.
Strong qualitative and quantitative research skills, including data analysis, uncertainty interpretation, and actionable insights into model behavior and performance.
Excellent problem-solving and stakeholder management skills, with the ability to navigate ambiguity, prioritize risks, balance speed with rigor, and influence cross-functional decisions.
Nice to have
Experience integrating automated evaluations and regression testing into engineering workflows, using tools such as Braintrust, LangSmith, DeepEval, or equivalent platforms.
Experience developing scalable, reusable evaluation frameworks and driving consistent quality standards across teams.
Background in product design, product management, data science, or engineering, with familiarity with Figma or similar design and developer tools.
Strong statistical expertise in sampling, significance testing, and agreement metrics, including experiments or performance-metric analysis for AI-powered features.
Advanced training in a quantitative or behavioral discipline through formal education or equivalent hands-on experience.
Culture & Benefits
Full-time senior individual contributor role with no direct reports, reporting to the AI Insights lead and working with AI Research within AI Platform.
Employees receive equity and benefits including health, dental, vision, retirement contributions, paid parental and family-planning leave, mental-health support, paid time off, sick leave, holidays, and flexible PTO for eligible exempt employees.
Additional benefits may include recharge days, cell phone and home internet reimbursements, and lifestyle spending accounts.
Video interviews require candidates to keep cameras on; in-person onboarding is required after hiring.
Figma is an equal opportunity workplace and provides reasonable accommodations throughout the application and employment process.
Hiring process
Video interviews require the candidate's camera to remain on.
Candidates may submit up to 5 applications during any rolling 30-day period.
Candidates may not reapply to the same role while an application is active or within 90 days after rejection.
Будьте осторожны
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →