AI Query Evaluation Specialist (AI Trainer)

TL;DR

AI Query Evaluation Specialist (AI Trainer): Evaluating and improving the quality of AI-powered search and assistant experiences with an accent on dataset curation and query annotation. Focus on benchmarking Microsoft Copilot against other leading AI systems by analyzing user intent and response quality.

Location: Remote (Must be based in the United States)

Salary: $80,000 – $95,000 USD annually

Company

A technology solutions firm headquartered in Bellevue, Washington, that leverages cutting-edge technology to help organizations create new revenue streams and business lines.

What you will do

• Analyze real user query logs to identify representative English-speaking market intent.
• Curate and maintain high-quality datasets for benchmarking Copilot, ChatGPT, and Gemini.
• Annotate queries based on dimensions such as web search necessity, PII presence, and domain expertise.
• Ensure annotation consistency and alignment with evolving evaluation guidelines.
• Organize and summarize evaluation outputs using Microsoft Excel.
• Identify query patterns to provide feedback for improving dataset coverage and quality.

Requirements

• Strong English reading comprehension to interpret subtle differences in user intent.
• Experience with structured data or annotation workflows.
• Proficiency in Microsoft Excel or similar data analysis tools.
• Familiarity with modern AI tools (e.g., Copilot, ChatGPT, Gemini).
• Must be based in the United States (implied by 401k, FSA, and state-specific salary ranges).

Nice to have

• Prior experience in data annotation, content evaluation, or search relevance for AI products.
• Basic statistical or data analysis knowledge.
• Fluency in English plus at least one additional language: Japanese, Korean, French, Chinese, German, or Italian.

Culture & Benefits

• Comprehensive medical, dental, and vision coverage.
• 401k program and Flexible Spending Account (FSA).
• Competitive PTO offerings and Parental Leave.
• Opportunities for professional growth and development within a bold and agile environment.