Software Engineer (AI Benchmarking)

TL;DR

Software Engineer (AI Benchmarking): Building and maintaining an AI Benchmarking Hub to evaluate model capabilities with an accent on infrastructure development and integration with AI providers. Focus on designing new benchmarks, implementing evaluation frameworks like Inspect, and collaborating with researchers to provide rigorous insights into AI trends.

Location: Fully remote. While open to applicants from all time zones, there is a preference for candidates who can overlap with UTC–8 (Pacific Time) and UTC (Greenwich Mean Time). Candidates should be able to travel for 3 staff retreats per year.

Salary: $125,000–$200,000 USD.

Company

hirify.global is a research institute dedicated to investigating machine learning trends and the economic consequences of AI to inform policymakers and the public.

What you will do

• Implement and maintain AI benchmarks within the evaluation infrastructure using the Inspect library.
• Develop new benchmarks and prototype innovative ideas for evaluation projects.
• Collaborate with researchers and analysts to ensure evaluation data is accurate and insightful.
• Facilitate internal experiments and integrate new model releases into the benchmarking suite.
• Contribute high-quality, robust code to complex systems and infrastructure.

Requirements

• Professional level English proficiency is required.
• More than two years of professional experience building and maintaining complex systems.
• Ability to write robust, maintainable code and dive deep into existing infrastructure.
• Strong interest in providing independent, rigorous insights into AI capabilities.
• Ability to travel for three staff retreats per year.

Nice to have

• Hands-on experience running LLM evaluations.
• Familiarity with evaluation frameworks like Inspect.
• Background in AI domain expertise or cybersecurity.

Culture & Benefits

• Comprehensive global benefits program including health, life insurance, and pension plans.
• Generous paid time off with 30 days protected and unlimited personal/sick leave.
• Up to 6 months of parental leave for permanent staff.
• Flexible expense policy for equipment, productivity tools, and learning opportunities.
• Access to Berkeley, California office with paid meals and gym access.