Clinical AI Evaluation Specialist (AI)

TL;DR

Clinical AI Evaluation Specialist (AI): Defining quality standards and evaluation frameworks for clinical AI outputs with an accent on ABA principles and AI governance. Focus on designing automated monitoring, conducting structured reviews of LLM content, and ensuring alignment with ethical standards.

Location: Remote (Must be based in the US)

Salary: $90,000 - $115,000 USD

Company

A leading provider of autism and IDD care software enabling therapy providers and educators to scale care delivery through innovative technology.

What you will do

• Design evaluation frameworks, including acceptance criteria, test plans, and performance benchmarks for clinical AI products.
• Conduct structured sampling reviews of AI outputs to ensure alignment with ABA principles and BACB ethical standards.
• Implement monitoring automation, quality checks, and drift detection in collaboration with engineering teams.
• Develop governance artifacts such as system cards, monitoring playbooks, and risk assessment documentation.
• Collaborate with product and engineering teams to integrate governance considerations early in the development lifecycle.
• Contribute to AI literacy training and thought leadership through publications and conference presentations.

Requirements

• Board Certified Behavior Analyst (BCBA) in good standing.
• 2+ years of experience in QA, evaluation, or data analysis, preferably in healthcare or behavioral health.
• Practical experience with LLMs, generative AI, prompt engineering, and AI workflow design.
• Knowledge of statistical sampling methodologies and healthcare quality assurance frameworks.
• Familiarity with AI governance frameworks (e.g., NIST AI RMF, EU AI Act) and HIPAA compliance.
• Must be based in the US.

Culture & Benefits

• Comprehensive health benefits and 401(k) matching.
• Generous PTO and paid parental leave.
• Flexible work environment with a focus on impact and inclusion.
• Career development support and wellness programs.
• Opportunities to engage in community giving through CR Cares™.