Staff Applied Scientist (AI)

TL;DR

Staff Applied Scientist (AI): Building and optimizing the evaluation framework for an AI-native observability dashboard with an accent on GenAI agents, tool-selection accuracy, and retrieval relevance. Focus on creating scalable measurement systems for non-deterministic agent trajectories and ensuring high-quality AI-driven insights.

Location: Hybrid in New York, NY, USA

Salary: $276,000 — $345,000 USD

Company

Datadog is a leading observability platform providing a unified single-pane-of-glass for metrics, logs, and traces.

What you will do

• Own the evaluation strategy for AI-native dashboards, defining offline and online metrics for quality and cost.
• Build evaluation datasets, golden traces, and regression harnesses to catch quality changes before production.
• Drive measurable improvements in retrieval relevance, tool-selection accuracy, and context efficiency.
• Partner with engineers to optimize single-turn and multi-turn agent success.
• Provide technical leadership across the organization through design reviews, working groups, and mentorship.

Requirements

• BS/MS/PhD in a scientific field or equivalent experience.
• 10+ years of relevant engineering or applied science experience, including experience as a technical lead.
• Proven track record of leading ML or GenAI initiatives from research through production in a product-driven environment.
• Significant experience with evaluation, experimentation, or measurement of ML systems at scale.
• Must be based in New York, NY, USA to support the hybrid work model.
• Strong product mindset and ability to drive initiatives across cross-functional teams.

Culture & Benefits

• New hire stock equity (RSUs) and employee stock purchase plan (ESPP).
• Comprehensive healthcare, dental, parental planning, and mental health benefits.
• 401(k) plan with company match.
• Continuous professional development, product training, and career pathing.
• Inclusive company culture with Community Guilds and global Spring Health benefits.
• Paid time off and fitness reimbursements.