Research Evaluation Specialist (AI)

Location: Remote, worldwide

Salary: $20–$70 per hour; output-based compensation is provided for each task meeting project specifications and quality requirements.

Company

Consulting work supporting the development and evaluation of advanced AI models.

What you will do

• Develop original, high-difficulty question-and-answer pairs that assess advanced AI model capabilities within the relevant area of expertise.
• Research, verify, and document answers using primary sources, scholarly literature, authoritative references, citations, and supporting reasoning.
• Design questions requiring advanced reasoning, methodological understanding, and synthesis across multiple sources.
• Test questions against AI systems, identify insufficiently challenging items, and iteratively increase complexity while maintaining factual accuracy.
• Write precise, clear, logically defensible questions and answers, incorporate reviewer feedback, and follow project quality standards.
• Maintain research integrity, thorough documentation, and analytical rigor across all deliverables.

Requirements

• Previous experience in AI training, data labeling, or AI model evaluation is required.
• Advanced academic qualifications or substantial research experience in the relevant discipline.
• Strong research expertise, advanced subject-matter knowledge, and familiarity with primary academic literature.
• Exceptional analytical thinking, attention to detail, and strong written communication skills in English.
• Experience locating, evaluating, and cross-verifying information from authoritative sources.
• Ability to work independently, manage priorities, and deliver high-quality research in a remote environment.
• Ability to develop original, challenging, and methodologically sound questions assessing deep subject-matter understanding.

Nice to have

• Completed Master’s or PhD, active doctoral candidacy, or equivalent research experience as a specialist, researcher, assistant professor, or professor in the relevant discipline.

Culture & Benefits

• Remote, worldwide working arrangement.
• Contractor role of approximately 15 hours per week.
• Output-based compensation tied to tasks meeting project specifications and quality requirements.
• Minimum weekly submission requirements apply, with task duration varying by expertise and workflow.
• Selected contributors are expected to begin their first tasks within 24–48 hours after onboarding.

Hiring process

• Positions are typically filled within 48 hours.
• Selected candidates should be prepared to begin promptly.
• First tasks are expected to start within 24–48 hours of completing onboarding.