Member Of Technical Staff, LLM Evaluation (AI)

TL;DR

Member Of Technical Staff, LLM Evaluation (AI): Developing and implementing cutting-edge methodologies to evaluate Copilot's performance in real-world scenarios, with an accent on identifying failure modes, developing new evaluation methods, and implementing real-time performance signals. Focus on creating automated evaluation frameworks, building automated testing systems, and adapting state-of-the-art algorithms for production systems.

Location: Mountain View, United States. Employees are expected to work from a designated Microsoft office at least four days a week if they live within 50 miles (U.S.) of that location.

Salary: USD $188,000 – $331,200 per year (range applies to IC5/IC6 levels in San Francisco Bay Area and New York City metropolitan area).

Company

Microsoft’s mission is to empower every person and every organization on the planet to achieve more.

What you will do

• Measure the performance of Copilot, identify failure modes, and develop novel mitigation strategies using techniques like data mining, prompt engineering, LLM as a judge, and classifier training.
• Create and implement comprehensive evaluation frameworks across diverse scenarios, edge cases, and potential failure modes.
• Build automated testing systems, generalize solutions into repeatable frameworks, and write efficient code for model pipelines and intervention systems.
• Maintain a user-oriented perspective by understanding needs from user perspectives and validating approaches through user research.
• Track advances in research, identify relevant state-of-the-art techniques, and adapt algorithms to drive innovation in production systems.

Requirements

• Doctorate in Data Science or a related field with 5+ years, OR Master’s Degree with 7+ years, OR Bachelor’s Degree with 10+ years of data science experience.
• Experience managing structured and unstructured data, applying statistical techniques, and reporting results.
• Experience prompting and working with large language models.
• Experience writing production-quality Python code.

Nice to have

• Demonstrated interest in Responsible AI.

Culture & Benefits

• Work within a culture of inclusion, respect, integrity, and accountability.
• Equal opportunity employer committed to diversity.
• Assistance with religious accommodations and reasonable accommodations due to a disability during the application process.