Member of Technical Staff, LLM Evaluation (AI)

TL;DR

Member of Technical Staff, LLM Evaluation (AI): Building and implementing cutting-edge methodologies for evaluating Copilot's real-world performance with an accent on developing new LLM evaluation methods, training classifiers, and experimenting with data collection techniques. Focus on creative problem solving, building automated evaluation frameworks, and driving improvements in AI systems for millions of users.

Location: New York, United States. Employees are expected to work from a designated hirify.global office at least four days a week if living within 50 miles (U.S.) or 25 miles (non-U.S., country-specific) of that location, starting January 26, 2026.

Salary: USD $220,800 – $331,200 per year (for New York City metropolitan area)

Company

hirify.global is a corporation empowering individuals and organizations globally through innovative technology and a culture of inclusion.

What you will do

• Develop and implement cutting-edge methodologies to evaluate Copilot's real-world performance.
• Measure Copilot's performance, identify failure modes, and propose mitigation strategies including data mining and prompt engineering.
• Create and implement comprehensive evaluation frameworks across diverse scenarios and potential failure modes.
• Build automated testing systems, generalize solutions into repeatable frameworks, and write efficient code for model pipelines.
• Maintain a user-oriented perspective by understanding needs from user research and serving as a trusted advisor on AI matters.
• Track research advances, identify relevant state-of-the-art techniques, and adapt algorithms to drive innovation in production systems.

Requirements

• Doctorate in Data Science or related field with 5+ years of data-science experience, OR Master’s Degree with 7+ years, OR Bachelor’s Degree with 10+ years.
• Extensive experience managing structured and unstructured data, applying statistical techniques, and reporting results.
• Experience prompting and working with large language models.
• Experience writing production-quality Python code.
• Demonstrated interest in Responsible AI.

Culture & Benefits

• Work in a culture of inclusion built on values of respect, integrity, and accountability.
• Grow with a growth mindset, innovate to empower others, and collaborate to achieve shared goals.
• Opportunity for various benefits and compensation.
• Equal opportunity employer with consideration for religious and disability accommodations.