Intelligence Analyst (AI Evaluation)

Location: Remote

Salary: $30–$80 per hour

Company

hirify.global provides consulting services for projects focused on evaluating and improving next-generation AI systems.

What you will do

• Develop and map analytical cases to established taxonomies across multiple intelligence evaluation areas.
• Create complex test cases covering source evaluation, analytical rigor, and the expression of uncertainty.
• Assess and grade AI-generated responses for reasoning quality, source use, recognition of limitations, and practical utility.
• Identify unnecessary model refusals of legitimate analytical exploration and provide clear documentation and actionable feedback.
• Document edge cases, analytical disagreements, and ambiguous scenarios to improve evaluation methodologies.
• Collaborate with project leads to refine source-evaluation, confidence-level assessment, analytical-review, and benchmarking standards.

Requirements

• Demonstrated all-source intelligence analysis experience or deep expertise in a specialized analytical discipline, ideally in a civilian Intelligence Community environment.
• Strong proficiency in source evaluation, confidence-level assessment, and established intelligence analysis standards.
• Experience teaching, mentoring, or leading analytical reviews, with the ability to promote analytical rigor.
• Excellent written and verbal communication skills, emphasizing clarity, precision, and evidence-based reasoning.
• Ability to identify and explain nuanced model limitations, including unnecessary refusals.
• Familiarity with structured analytical techniques and intelligence tradecraft standards.

Nice to have

• Previous experience in AI evaluation, model assessment, or related quality assurance initiatives.

Culture & Benefits

• Remote contractor role.
• Opportunity to contribute to the accuracy, reliability, and consistency of next-generation AI systems.
• Hourly compensation of $30–$80.