Relevance Metrics Data or Applied Scientist (AI)

TL;DR

Relevance Metrics Data or Applied Scientist (AI): Designing and implementing metrics and quality control for the Bing Grounding API, focusing on RAG quality metrics within Bing Search and evaluating search results using LLMs. Focus on building complex data pipelines, prompt engineering for multi-step evaluations, and generating actionable insights across various teams.

Location: Mountain View, United States. This role is hybrid, requiring employees within a 50-mile commute of a designated Microsoft office in the U.S. to work from the office at least four days per week.

Salary: USD $119,800 – $234,700 per year (general U.S.) or USD $158,400 – $258,000 per year (San Francisco Bay area and New York City metropolitan area).

Company

hirify.global is a core team within Bing Search, focusing on powering billions of daily searches using AI and Large-Language Models.

What you will do

• Design and implement metrics for RAG with Bing Web Search and other APIs.
• Build pipelines and dashboards for Bing Grounding quality.
• Use LLM models in LLM-as-a-judge settings for data evaluation.
• Engineer prompts for textual and multi-model LLMs for data processing and insights.
• Design and implement E2E pipelines from sampling anomalies to auto-updatable dashboards.
• Apply classical ML along with LLM to augment data analysis and processing.
• Help teams to build new innovative search experience with Bing.

Requirements

• Doctorate with 1+ year, Master’s with 3+ years, or Bachelor’s with 5+ years of data-science experience.
• 3 years of T-SQL.
• 5+ years of C#, Python, Java or any other major programming language.
• Experience with Large Language Models.
• Work from office required at least four days per week for employees within 50-mile commute of a U.S. Microsoft office.

Nice to have

• Doctorate with 3+ years, Master’s with 5+ years, or Bachelor’s with 7+ years of data-science experience.
• Experience or deep interest in RAG.
• Passion for metrics building for complex multi-step systems.
• Passion for prompt engineering and text generations with LLMs.
• Interest in designing dashboards for data visualization in novel ways.
• Experience with Machine Learning.
• Solid collaboration and communication skills.

Culture & Benefits

• Opportunity to work with multiple teams across Bing (80+ teams) with high visibility and impact.
• Culture of inclusion, growth mindset, innovation, respect, integrity, and accountability.
• Comprehensive benefits and pay information available on Microsoft Careers website.