AI Evaluation & Quality Assurance Specialist

Location: Remote within Argentina, Brazil, Mexico, Uruguay, Peru, or Chile; 8 hours of mandatory PST overlap required.

Company

hirify.global is a boutique consultancy providing IT professional services and engineering talent solutions.

What you will do

• Validate task instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
• Review AI agent execution traces, tool calls, and deliverables to determine whether outcomes are justified.
• Audit grading logic for brittle checks, incorrect expected answers, unsupported rubric criteria, and unfair penalties.
• Investigate discrepancies between model performance, grader results, and expected outcomes.
• Distinguish model limitations from task defects, grader errors, and environment or tool failures.
• Document evidence-based findings, flag uncertainty, and verify that revisions resolve identified issues.

Requirements

• Comfort reading and interpreting Python, SQL, shell scripts, structured data, and execution logs.
• Experience reviewing technical workflows, software behavior, data outputs, or evaluation logic.
• Strong analytical skills, including verifying calculations and reconciling conflicting evidence.
• Ability to assess the correctness and completeness of technical deliverables.
• Strong written English required, with experience providing clear, specific, and reproducible feedback.
• High attention to detail when identifying inconsistencies, missing information, and evaluation defects.

Culture & Benefits

• Remote hourly contractor engagement.
• 40 hours per week for a 10-week duration.
• Immediate start.