Senior Data Scientist AI Evaluation

Location: Remote - Americas

Company

hirify.global is a US-headquartered brokerage infrastructure company providing APIs and financial services for stocks, ETFs, options, crypto, fixed income, and 24/5 trading.

What you will do

• Design AI evaluations by defining ground truth, metrics, and scoring methods for models and agents.
• Build repeatable evaluation loops that track quality over time and detect regressions before release.
• Partner with Engineering and Analytics Engineering to operationalize evaluation harnesses.
• Translate evaluation results into actionable recommendations for system improvements.
• Establish evaluation guidelines, documentation, quality standards, and review practices.
• Mentor colleagues and promote measurable AI quality across the organization.

Requirements

• Track record of rigorous quantitative measurement, such as model evaluation, metric validation, or experimentation.
• Strong statistical and machine learning foundation, including sample sizing, confidence intervals, significance testing, non-determinism, and validation of automated graders against human ground truth.
• Proficiency in Python and SQL, with experience evaluating models in production environments.
• Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
• Excellent communication, cross-functional collaboration, and problem-solving skills.
• 6–10 years of experience in quantitative data science or machine learning, with focused experience in measurement or evaluation.

Nice to have

• Production experience with LLM or agent evaluation, evaluation harnesses, LLM-as-judge calibration, and CI regression gates.
• Experience evaluating text-to-SQL, analytics agents, or systems whose correctness can be verified against data.
• Fintech, brokerage, or other experience involving business or risk consequences from incorrect answers.
• Fluency with AI tools used in research and engineering workflows.

Culture & Benefits

• Work with a globally distributed team of more than 400 people.
• Health benefits and stock options.
• One-time USD $500 home-office setup allowance for new hires.
• USD $150 monthly stipend via a Brex Card.
• Values include curiosity, empathy, and accountability.