Senior Data Scientist - LLM Evaluation (AI)

TL;DR

Senior Data Scientist (AI): Architecting the LLM evaluation framework to define rigorous statistical standards that models must meet before production. Focus on turning subjective outputs into objective, measurable data and ensuring AI products are safe, accurate, and reliable.

Salary: $200-240K

Company

hirify.global delivers state of the art technology that helps insurance claims teams make claims handling more accurate, fair, and efficient.

What you will do

• Design and implement comprehensive scorecards and benchmarking suites for LLM-based extraction, summarization, and chat interfaces.
• Work with Subject Matter Experts (SMEs) to codify their expertise into evaluation datasets and "ground truth" labels.
• Design the statistical guardrails to scale both human and automated labeling efforts.
• Provide clear, data-driven "Go/No-Go" recommendations for model deployment based on error analysis and statistical confidence intervals.

Requirements

• 5+ years of experience in Data Science with a strong background in traditional statistics.
• 2+ years of focused experience working with LLMs, specifically in evaluation, benchmarking, and prompt auditing.
• Master’s or PhD in Statistics, Mathematics, or a related quantitative field.
• Proven ability to work with non-technical SMEs to translate their qualitative feedback into quantitative metrics.
• Proficient in Python (Pandas, Scikit-learn, Statsmodels) and SQL.

Nice to have

• Deep knowledge of metrics like Cohen’s Kappa or Fleiss' Kappa to quantify agreement between SMEs and evaluate the clarity of labeling instructions.
• Experience in Active Learning
• Experience with platforms like Labelbox, Snorkel, or Prodigy to manage the flow between human annotators and automated systems.

Culture & Benefits

• Medical, dental, vision, short & long-term disability, life insurance and AD&D, and 401k matching.
• Paid time off and sick leave, 100% paid parental leave.
• Flexible schedule for new parents returning to work.
• Catered lunches, happy hours, pet-friendly spaces, and monthly technology stipend.
• $1,000/year for each employee for professional development, as well opportunities for tuition reimbursement.

Hiring process

• We are open to sponsoring candidates currently in the U.S. who need to transfer their active visa.