Senior Data Scientist, AI Scoring & Evaluation (AI)

Location: East

Company

hirify.global is a Silicon Valley skills intelligence platform using proprietary AI agents to help organizations understand, mobilize, manage, and develop talent.

What you will do

• Own scoring quality end to end, including evaluator design, rubric anchoring, calibration, and accuracy against expert benchmarks.
• Build and operate continuous evaluation harnesses with gold sets, bias diagnostics, drift detection, and pre-release quality gates.
• Define and publish assessment quality KPIs, including human-to-AI agreement, reliability, classification accuracy, bias indicators, latency, and cost per assessment.
• Lead performance studies, impact simulations, root-cause investigations, and analytics that inform assessment and product decisions.
• Implement and coordinate measurement improvements with assessment technology and data engineering, owning specifications, validation, and quality standards.
• Support AI governance and InfoSec by translating frameworks such as GDPR and the EU AI Act into evidence for audits, security reviews, and enterprise diligence.

Requirements

• Senior-level production data science experience with independent end-to-end ownership of data projects.
• At least 4 years of experience, or 3 years with demonstrated end-to-end ownership, ideally in an AI startup or fast-moving product environment.
• Strong Python and SQL skills, with the ability to conduct analyses without engineering support.
• Advanced statistical and analytical skills, including reliability studies, performance analysis, and defensible evaluation methods.
• Production experience with LLM-based systems, including prompt design and evaluation against human judgment.
• Ability to explain quantitative results clearly to executives, customers, and other non-technical audiences.

Nice to have

• Experience with educational, learning, or assessment data.
• Background in psychometrics or educational measurement.

Culture & Benefits

• AI assistants, copilots, and automation tools are expected to be part of daily work.
• Teams are encouraged to automate workflows and build repeatable, scalable systems.
• The role operates in a small, senior, cross-functional environment spanning Engineering, Assessment Science, Design, Product, and Data Engineering.
• hirify.global is backed by NEA, Jump Capital, and Owl Ventures and serves enterprise clients including Accenture, Siemens Energy, Samsung, and the United States Air Force.