Machine Learning Scientist (AI)

Location: SF Bay Area; on-site

Company

hirify.global is an AI evaluation platform that measures how frontier models perform in real-world workflows using human preference signals and open datasets.

What you will do

• Design and conduct experiments evaluating AI model behavior across reasoning, style, robustness, and user preference dimensions.
• Develop metrics, methodologies, and evaluation protocols beyond traditional benchmarks.
• Analyze large-scale human voting and interaction data to identify insights into model performance and user preferences.
• Collaborate with engineers to implement and scale research findings into production systems.
• Prototype research ideas, author internal reports and external publications, and contribute to the transparency of the hirify.global Intelligence leaderboard and tools.
• Partner with model providers on evaluation questions and responsible model testing.

Requirements

• PhD or equivalent research experience in Machine Learning, Natural Language Processing, Statistics, or a related field.
• Hands-on experience training large-scale models, including reward models, preference models, and fine-tuning LLMs with RLHF, DPO, or contrastive learning.
• Strong foundation in machine learning and statistics, including rigorous experimental design, evaluation, and ablation.
• Strong understanding of LLMs and modern deep learning architectures, including Transformers, diffusion models, and reinforcement learning with human feedback.
• Proficiency in Python and ML research libraries such as PyTorch, JAX, or TensorFlow.
• Experience publishing research or working on open-source ML, NLP, or AI evaluation projects, with the ability to collaborate across engineering and product teams.

Culture & Benefits

• Work on AI evaluation and research with a small, mission-driven team.
• Collaborate with researchers, academics, engineers, product teams, marketing, and the broader research community.
• Comprehensive medical, dental, vision, and wellness benefits.
• Competitive compensation and equity aligned with the candidate's permanent work location.
• Culture focused on transparency, trust, community impact, open science, and reproducibility.