Researcher (AI)

TL;DR

Researcher (AI): Leading the evaluation of frontier AI models on complex, real-world office tasks with an accent on designing robust grading rubrics and benchmarking methodologies. Focus on assessing practical model capabilities, automating evaluation workflows, and communicating research findings through public-facing reports.

Location: Remote (Global). Preference for candidates who can overlap with UTC–8 (Pacific Time) and UTC (Greenwich Mean Time).

Salary: $115,000 – $200,000 USD per year.

Company

hirify.global is a research institute investigating machine learning trends and the economic consequences of AI to inform policymakers and industry leaders.

What you will do

• Create and curate an evaluation suite of challenging, real-world tasks for frontier AI models.
• Design and refine grading rubrics to assess AI performance both quantitatively and qualitatively.
• Regularly evaluate new AI models and products against the established task suite.
• Analyze evaluation results and compare model performance across different tasks.
• Communicate research findings through public-facing reports, blog posts, and data visualizations.
• Automate parts of the evaluation workflow and develop standalone benchmarks.

Requirements

• Professional level English proficiency required.
• Strong analytical thinking and experience conducting rigorous experiments.
• Grounded, skeptical mentality regarding AI capabilities versus marketing hype.
• Experience working with AI agents and tools.
• Familiarity with existing AI benchmarks and evaluation methodologies.
• Comfort with data analysis and light coding to process research results.
• Ability to travel for three staff retreats per year.

Nice to have

• Experience testing frontier models and writing capability assessments.
• Proficiency in Python.

Culture & Benefits

• Fully remote environment with flexible work hours.
• Competitive global benefits program including health, life insurance, and pension plans.
• Generous PTO policy with 30 days protected, unlimited personal/sick leave, and 4 months paid parental leave.
• Flexible expense policy for equipment, productivity tools, and AI subscriptions.
• Paid work trips for staff retreats and relevant conferences.
• Access to Berkeley, California office with gym and meals for all staff.