Expert Reviewer (Healthcare AI)

Location: Remote

Company

hirify.global provides independent, multidisciplinary evaluations of healthcare AI technologies seeking deployment in real-world clinical settings, including work supporting Utah's Office of Artificial Intelligence Policy regulatory sandbox.

What you will do

• Evaluate model performance, validation methods, reference standards, sample sizes, statistical analyses, performance metrics, operating thresholds, and reported uncertainty.
• Assess data quality, representativeness, data leakage, subgroup performance, and generalizability across patient populations and healthcare settings.
• Evaluate technical reliability, robustness, reproducibility, failure modes, out-of-distribution performance, monitoring plans, change management, and rollback criteria.
• Review proposed evaluation plans, technical safeguards, escalation mechanisms, and human-in-the-loop processes for pilot deployment.
• Collaborate with clinical and technical reviewers to identify cross-disciplinary concerns and document strengths, limitations, missing evidence, and technical findings.
• Provide evidence-based recommendations, participate in reviewer calibration and quality assurance, and maintain independence, confidentiality, and appropriate data handling.

Requirements

• At least 3 years of professional or applied research experience in machine learning, data science, biostatistics, clinical informatics, or a closely related field, with direct experience evaluating or validating predictive models or AI systems.
• Understanding of model evaluation methodologies, including performance metrics, validation design, statistical uncertainty, threshold selection, and common sources of bias or methodological error.
• Experience assessing model performance on independent or held-out data and interpreting whether reported performance generalizes beyond the evaluation dataset.
• Hands-on experience evaluating at least one relevant AI system class, such as clinical natural language processing, large language models, conversational AI, medical imaging, computer vision, or tabular predictive models.
• Ability to critically assess technical research, evaluation reports, or validation studies and identify methodological limitations or unsupported conclusions.
• Strong analytical writing skills and the ability to communicate complex findings clearly to non-technical audiences.
• Ability to work independently while contributing to multidisciplinary expert review and calibration processes.
• Willingness to complete confidentiality agreements, conflict-of-interest disclosures, and reviewer training requirements.

Nice to have

• Experience evaluating AI or ML systems used in healthcare, clinical decision support, digital health, or other high-stakes environments.
• Familiarity with TRIPOD+AI, STARD-AI, CONSORT-AI, DECIDE-AI, model cards, or similar clinical AI evaluation and reporting frameworks.
• Experience evaluating generative AI, LLM-based systems, or autonomous AI agents, including multi-turn behavior, reliability, hallucinations, and technical safeguards.
• Experience with robustness testing, adversarial evaluation, red teaming, or systematic failure-mode analysis.
• Knowledge of production monitoring, distribution shift, model drift, version control, and change-management practices.
• Experience designing or evaluating human-in-the-loop systems or comparing model performance against human expert judgments.
• Familiarity with FDA guidance for AI-enabled medical devices, clinical decision support, or software as a medical device.
• Experience conducting independent technical reviews, scientific peer reviews, grant reviews, or formal technology evaluations.
• Experience collaborating with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment.

Culture & Benefits

• Remote, contract-based engagement with assignments made according to technical expertise and the AI systems under evaluation.
• Per-review assignments include reviewing applicant materials, documenting findings within a standardized evaluation framework, and participating in review discussions or calibration activities as needed.
• Orientation to the evaluation framework and project-specific requirements is provided before the first assignment.
• Work emphasizes independent, evidence-based review, multidisciplinary collaboration, confidentiality, and reviewer quality assurance.