AI Research Engineer

Location: Remote - US; candidates must be based in the Bay Area. hirify.global cannot hire in Montana, Hawaii, Alaska, or Washington DC.

Compensation: $250K–$400K OTE + equity.

Company

hirify.global provides a UX research platform and an AI Training & Evals unit that develops human data, evaluations, and benchmarks for measuring AI performance with real people in real-world contexts.

What you will do

• Design and validate rubrics for validity, reliability, and sensitivity, including human-rater reliability and disagreement analysis.
• Calibrate AI judges against human judgment and investigate drift and noise in evaluation signals.
• Design hirify.global’s first long-horizon benchmark using real users, real tasks, and multi-week contextual studies.
• Define benchmark protocols, task design, scoring, release processes, and methods for monitoring drift, contamination, and difficulty.
• Study human feedback, preference data, human-AI collaboration, user simulations, and behavioral evaluation of agents.
• Build research prototypes and methods that can run in live customer programs, write papers, and collaborate with external labs and academics.

Requirements

• Published research, strong preprints, or workshop papers in evaluations, preference learning, reward modeling, human-AI interaction, measurement, or agent evaluation.
• A PhD, a PhD in progress, or an MS with two or more strong research internships at labs or top industry groups; creative machine learning research experience without an advanced degree is also considered.
• Strong applied statistics, including inter-rater reliability, confidence intervals, statistical power, and identifying misleading metrics.
• Hands-on experience with LLM-as-judge, rubric-based evaluation, and preference data.
• Strong Python and the ability to build research prototypes and evaluation pipelines used by others.
• Experience running experiments on messy human data rather than only curated datasets.
• Clear technical writing and the ability to explain research to lab researchers and product managers.
• Must be based in the Bay Area.

Nice to have

• Long-horizon or agentic evaluation experience.
• Psychometrics, item response theory, or human-subjects research experience.
• Open-source benchmark or dataset contributions.

Culture & Benefits

• Remote-first work environment with employees able to work from almost anywhere in the United States, subject to location exclusions.
• Flexible PTO, 15 company holidays, and 12 weeks of paid parental leave.
• 401(k) match, education stipend, conference budget, and remote work stipend.
• Publication freedom, built-in bonus and equity program, and transparent, diverse, inclusive working culture.

Hiring process

• Conversation with the Head of AI Research.
• Research talk on a project led by the candidate.
• Design exercise on validating a rubric for a long-horizon task, followed by conversations with the GM, forward deployed engineers, and delivery partners.