TL;DR

Sr Lead Machine Learning Engineer (AI): Designing and implementing comprehensive evaluation frameworks for AI agents and LLM-based systems with an accent on human+AI collaboration outcomes and robust benchmarking pipelines. Focus on translating complex signals into actionable insights, influencing product roadmaps, and championing rigorous experimental design.

Location: Remote (US)

Salary: $195,000 - $308,000 USD

Company

%hirify_global% is the world’s human and AI-powered work marketplace connecting businesses with skilled, AI-enabled independent talent globally.

What you will do

Design and implement comprehensive evaluation frameworks for agentic AI systems focusing on human+AI collaboration.
Build benchmarking pipelines to capture nuanced success indicators like trust calibration and intervention frequency.
Lead development of observability tools and instrumentation for analyzing agent behavior in production.
Translate complex qualitative and quantitative signals into actionable insights that inform model iteration and product prioritization.
Collaborate with researchers, engineers, and product teams to align evaluation methodologies with business and user goals.
Own benchmarking infrastructure that enables reproducible, scalable evaluation across AI initiatives.

Requirements

Proven experience designing evaluation systems for agentic or LLM-based AI, ideally in complex, interactive or open-ended environments.
Deep expertise in statistical experimentation, benchmark creation, and human-AI interaction assessment.
Fluency in building data pipelines and tooling using Python, SQL, and distributed data processing frameworks.
Demonstrated ability to influence product and model roadmaps through evaluation insights and performance measurement.
Adaptive-level proficiency in integrating AI tools into technical workflows for analysis, experimentation, and observability refinement.
Must be a full-time employee in one of the 34 U.S. states where %hirify_global% hires.

Culture & Benefits

Shape the future of work for a global, remote-first workforce, creating economic opportunities worldwide.
Culture built on trust, risk-taking, customer focus, and excellence.
Comprehensive medical coverage for you and your family.
Unlimited PTO and a 401(k) plan with matching.
12 weeks of paid parental leave and an Employee Stock Purchase Plan.
Emphasis on personal and professional growth through development programs and mentorship.