Data Annotation Specialist (AI)

TL;DR

Data Annotation Specialist (AI): Evaluating, stress-testing, and improving Large Language Models through structured feedback and data labeling with an accent on analytical judgment and quality assurance. Focus on identifying model failure modes, creating high-quality training datasets, and maintaining consistency across complex multimodal tasks.

Location: Must be based in Canada

Salary: CAD $30/hour

Company

A leading security-first enterprise AI company building cutting-edge foundation models and end-to-end products to solve real-world business problems.

What you will do

• Evaluate and rank model outputs based on accuracy, helpfulness, and safety.
• Stress-test models to surface failure modes and unsafe behaviors.
• Author high-quality prompts and responses to build training datasets.
• Design and apply grading rubrics to ensure consistent annotation quality.
• Annotate and rectify inaccuracies across text, image, and structured data formats.
• Communicate performance trends and quality issues to cross-functional teams.

Requirements

• Must be based in Canada
• 1+ years of experience in AI data annotation, LLM evaluation, or related analytical roles.
• Strong command of written English with the ability to provide clear justifications for evaluations.
• Comfort working with structured data formats including JSON, CSV, Markdown, XML, and YAML.
• Ability to commit to 16 hours per week.
• Must provide own laptop (BYOD).

Culture & Benefits

• Opportunity to contribute to frontier AI technology development.
• Flexible, remote-first work environment.
• Exposure to experimental and evolving AI task types.
• Collaborative work within a global, asynchronous team.

Hiring process

• Resume and writing sample review.
• Virtual annotation test covering language-based tasks.
• Video screen with the Operations Team.