Member of Technical Staff, LLM Evaluation (AI)

TL;DR

Member of Technical Staff, LLM Evaluation (AI): Developing and implementing cutting-edge methodologies to evaluate Copilot's performance in real-world scenarios with an accent on developing new methods for LLM evaluation, classifier training, and data collection. Focus on building automated evaluation frameworks to drive improvements in Copilot, navigating complexity, and adapting state-of-the-art techniques.

Location: Mountain View, United States. Employees are expected to work from a designated Microsoft office at least four days a week if they live within 50 miles (U.S.) of that location.

Salary: USD $119,800 – $304,200 per year

Company

hirify.global is a global technology corporation focused on empowering individuals and organizations through innovative AI solutions.

What you will do

• Measure Copilot performance, identify failure modes, and develop mitigation strategies using data mining, prompt engineering, LLM as a judge, and classifier training.
• Create and implement comprehensive evaluation frameworks across diverse scenarios and edge cases.
• Build automated testing systems and generalize solutions into repeatable frameworks.
• Write efficient code for model pipelines and intervention systems.
• Maintain a user-oriented perspective by validating approaches through user research.
• Track research advances and adapt state-of-the-art techniques to drive innovation in production systems.

Requirements

• Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience OR Master’s Degree with 3+ years OR Bachelor’s Degree with 5+ years of data-science experience.
• Experience prompting and working with large language models.
• Experience writing production-quality Python code.

Culture & Benefits

• Work for Microsoft, a corporation committed to empowering every person and organization.
• Join a culture of inclusion built on values of respect, integrity, and accountability.
• Opportunity to innovate and collaborate to achieve shared goals.
• Access to comprehensive benefits and compensation information.