Quality Analyst II (AI)

TL;DR

Quality Analyst II (AI): Evaluating and optimizing AI-powered workflows for customer success advisors with an accent on AI output quality, prompt refinement, and failure mode classification. Focus on closing the loop between error detection and fix implementation within the internal AI platform ORA.

Location: Remote (United States)

Company

AI-powered business operating system providing infrastructure for agencies and SMBs to build, automate, and scale.

What you will do

• Review and score AI output across the call lifecycle, including pre-call briefs, in-call guidance, and post-call summaries.
• Classify failure modes such as hallucinations, missing context, or tone mismatches using standardized rubrics.
• Refine prompts and update knowledge base entries within the ORA platform to improve output quality.
• Analyze and surface systemic trends and quality gaps to leadership and the Lead Analyst.
• Partner with RevOps Engineering to co-define quality criteria for new AI skills and orchestrations.
• Stay current on LLM behavior and run experiments with new prompts and evaluation techniques.

Requirements

• Must be based in the United States.
• 2+ years of experience in Quality Assurance, Customer Support, or AI evaluation.
• Working understanding of LLM behavior, prompt design, and conversational AI patterns.
• Comfort reading code and making minor edits to debug prompt behavior and integrations.
• Strong written communication skills with the ability to document failure modes clearly for engineering.
• Bachelor’s degree in a related field or equivalent practical experience.

Nice to have

• Hands-on experience with AI development platforms such as Claude or Cursor.
• Familiarity with retrieval-augmented generation (RAG), vector databases, or SQL.
• Experience with APIs, webhooks, or data integration between systems.
• Background in customer success, support, or call center environments.
• Prior experience evaluating AI-generated conversations or chatbots.

Culture & Benefits

• Remote-first organization built for speed, ownership, and initiative.
• Culture that values innovation, clarity, and people-first approach.
• Deliberate growth path from an evaluator-first role to deeper AI creation work.