Senior Software Engineer (AI Eval)

TL;DR

Senior Software Engineer (AI Eval): Building evaluation infrastructure to measure accuracy, reliability, and performance of AI systems with an accent on designing datasets, benchmarks, and automated test harnesses. Focus on translating ambiguous AI behavior into measurable signals to ensure correct, safe, and predictable system scaling.

Location: Hybrid work model in San Francisco, California (Mondays, Tuesdays, and Thursdays are in-office anchor days).

Salary: $240,000–$280,000

Company

hirify.global builds performance and error monitoring tools to help developers write better software faster.

What you will do

• Design and build robust evaluation frameworks to measure accuracy, reliability, and regressions in AI systems.
• Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data.
• Build automated test harnesses and metrics pipelines for continuous evaluation of models and agentic workflows.
• Partner with applied AI engineers and product leaders to define and translate AI performance into measurable criteria.
• Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring.

Requirements

• Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field.
• Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred).
• Comfort writing production-quality code (Python and TypeScript).
• Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines.
• Familiarity with modern ML systems and evaluation techniques.

Nice to have

• Experience evaluating LLMs, agentic systems, or AI-assisted developer tools.

Culture & Benefits

• Hybrid work model with in-office anchor days to encourage collaboration.
• Mission to help developers write better software faster and enjoy technology.
• Commitment to providing equal employment opportunities and building a diverse, inclusive team.
• Eligibility to participate in employee benefit plans including incentive compensation, equity grants, paid time off, and group health insurance coverage.