Staff Applied Researcher, AI Quality (AI)

TL;DR

Staff Applied Researcher, AI Quality (AI): Designing evaluation systems and influencing how millions of developers experience AI with an accent on Large Language Model (LLM) evaluation and LLM agents. Focus on building scalable automatic metrics, LLM‑judge systems, reward models, and human‑in‑the‑loop evaluation pipelines.

Location: Remote, United States

Salary: USD $140,400.00 - USD $372,300.00 /Yr

Company

hirify.global is the world’s leading platform for agentic software development — powered by Copilot to build, scale, and deliver secure software.

What you will do

• Design next‑generation evaluation frameworks for code generation, reasoning, safety, multimodal tasks, and agentic workflows.
• Develop scalable automatic metrics, LLM‑judge systems, reward models, and human‑in‑the‑loop evaluation pipelines.
• Build and optimize evaluation tooling, datasets, benchmarking systems, and experimentation pipelines.
• Shape hirify.global’s strategy for model quality, alignment, and evaluation.
• Mentor other researchers and engineers, helping elevate technical standards across the organization.

Requirements

• Bachelor's degree in Data Science, Mathematics, Physics, Statistics, Economics, Operations Research, Computer Science, or related field AND 8+ years' experience in data science or related field, OR master's degree AND 6+ years' experience, OR doctorate AND 4+ years' experience, OR equivalent experience.
• 3+ years of strong engineering skills in Python/Typescript and experience building production grade evaluation or data/ML pipelines at scale.
• Proven track record shipping research or evaluation systems in production environments.
• Strong cross‑functional communication and influence skills.

Nice to have

• Experience with LLM judge systems, reward modeling, alignment, or safety evaluations.
• Background in code generation, developer tools, or AI‑assisted programming.
• Experience with large‑scale experimentation and online/offline evaluation strategies.
• Open‑source contributions or experience working with developer communities.
• Experience designing and leading complex research projects from ideation to implementation

Culture & Benefits

• Remote-first company.
• Competitive pay and generous learning and growth opportunities.
• Excellent benefits to support you, wherever you are.
• Diverse and inclusive environment.