Staff Applied Scientist (AI)

TL;DR

Staff Applied Scientist (AI): Developing and optimizing evaluation systems for AI agent interfaces to measure answer quality, tool-selection accuracy, and end-to-end success with an accent on measurement systems and dataset construction. Focus on solving non-deterministic trajectory evaluation and scaling measurement harnesses across multiple agent integrations.

Location: Onsite in New York, New York, USA

Salary: $276,000 — $345,000 USD

Company

hirify.global is the leading observability and security platform for the AI era, providing businesses with unified visibility across the technology stack to manage complexity at scale.

What you will do

• Define and own the evaluation strategy for AI agent integrations, including offline/online quality and cost metrics.
• Build evaluation datasets, golden traces, and regression harnesses to ensure quality before customer release.
• Improve retrieval relevance, tool-selection accuracy, and context efficiency in partnership with AI engineers.
• Conduct applied research on tool selection in large catalogs and multi-turn agent evaluation.
• Collaborate with first-party agent teams (Bits SRE, Assistant, Dev Agent) to standardize measurement.
• Provide technical leadership via design reviews and mentorship, and represent the company in the open agent ecosystem.

Requirements

• BS/MS/PhD in a scientific field or equivalent experience.
• 10+ years of relevant engineering or applied science experience, including technical lead roles.
• Proven track record of leading ML or GenAI initiatives from research to production.
• Significant experience in the measurement and experimentation of ML systems at scale.
• Strong product mindset and ability to lead cross-functional initiatives.
• Ability to operate effectively in ambiguous environments.

Culture & Benefits

• Competitive salary and equity package including RSUs and ESPP.
• Comprehensive healthcare, dental, and mental health benefits.
• 401(k) plan with company match.
• Continuous professional development, product training, and career pathing.
• Inclusive culture with Community Guilds and giving programs.