Member of Technical Staff, LLM Evaluation (AI)

TL;DR

Member of Technical Staff, LLM Evaluation (AI): Developing and implementing cutting-edge methodologies to evaluate Copilot's performance in real-world scenarios with an accent on developing new LLM evaluation methods, training classifiers, and experimenting with data collection techniques. Focus on creating comprehensive evaluation frameworks and building automated testing systems for AI systems.

Location: Boulder, United States. Employees are expected to work from a designated Microsoft office at least four days a week if they live within 50 miles.

Salary: USD $119,800 – $274,800 per year (U.S. general range, for IC4-IC5 levels); USD $158,400 – $304,200 per year (San Francisco Bay Area and New York City metropolitan area, for IC4-IC5 levels).

Company

hirify.global aims to empower every person and organization to achieve more by building and improving innovative AI products like Copilot.

What you will do

• Measure Copilot's performance and identify failure modes, leveraging data mining, prompt engineering, LLM as a judge, and classifier training.
• Solve complex problems, independently shaping direction and delivering results.
• Create and implement comprehensive evaluation frameworks across diverse scenarios and potential failure modes.
• Build automated testing systems, generalize solutions, and write efficient code for model pipelines.
• Maintain a user-oriented perspective by understanding needs and validating approaches through user research.
• Track research advances, identify state-of-the-art techniques, and adapt algorithms to drive innovation.

Requirements

• Bachelor’s Degree in Computer Science, Statistics, Economics, Psychology, Linguistics or related technical discipline.
• 4+ years of technical engineering experience with coding in Python and SQL.
• Experience prompting and working with large language models.
• Experience writing production-quality Python code.

Nice to have

• Demonstrated interest in Responsible AI.

Culture & Benefits

• Foster a growth mindset and collaborate to achieve shared goals.
• Uphold values of respect, integrity, and accountability.
• Contribute to a culture of inclusion where everyone can thrive.
• Access to additional benefits and compensation information via Microsoft's careers portal.

Hiring process

• Microsoft accepts applications and processes offers on an ongoing basis.