Sr Lead Machine Learning Engineer (AI)

TL;DR

Sr Lead Machine Learning Engineer (AI): Designing and implementing comprehensive evaluation frameworks for AI agents and LLM-based systems with an accent on human+AI collaboration outcomes and robust benchmarking pipelines. Focus on translating complex signals into actionable insights, influencing product roadmaps, and championing rigorous experimental design.

Location: Remote (US)

Salary: $195,000 - $308,000 USD

Company

hirify.global is the world’s human and AI-powered work marketplace connecting businesses with skilled, AI-enabled independent talent globally.

What you will do

• Design and implement comprehensive evaluation frameworks for agentic AI systems focusing on human+AI collaboration.
• Build benchmarking pipelines to capture nuanced success indicators like trust calibration and intervention frequency.
• Lead development of observability tools and instrumentation for analyzing agent behavior in production.
• Translate complex qualitative and quantitative signals into actionable insights that inform model iteration and product prioritization.
• Collaborate with researchers, engineers, and product teams to align evaluation methodologies with business and user goals.
• Own benchmarking infrastructure that enables reproducible, scalable evaluation across AI initiatives.

Requirements

• Proven experience designing evaluation systems for agentic or LLM-based AI, ideally in complex, interactive or open-ended environments.
• Deep expertise in statistical experimentation, benchmark creation, and human-AI interaction assessment.
• Fluency in building data pipelines and tooling using Python, SQL, and distributed data processing frameworks.
• Demonstrated ability to influence product and model roadmaps through evaluation insights and performance measurement.
• Adaptive-level proficiency in integrating AI tools into technical workflows for analysis, experimentation, and observability refinement.
• Must be a full-time employee in one of the 34 U.S. states where hirify.global hires.

Culture & Benefits

• Shape the future of work for a global, remote-first workforce, creating economic opportunities worldwide.
• Culture built on trust, risk-taking, customer focus, and excellence.
• Comprehensive medical coverage for you and your family.
• Unlimited PTO and a 401(k) plan with matching.
• 12 weeks of paid parental leave and an Employee Stock Purchase Plan.
• Emphasis on personal and professional growth through development programs and mentorship.