Senior Machine Learning Engineer (Agent Oversight)

TL;DR

Senior Machine Learning Engineer (Agent Oversight) (LLMs/Agentic AI): Building production ML systems that observe, evaluate, and improve agent behavior with an accent on evaluation frameworks, observability, and reliable improvement loops. Focus on detecting drift and misalignment, designing rigorous experiments, and taking novel methods from prototype to scalable production systems.

Location: San Francisco, CA or New York, NY. The position is listed for the United States; the stated salary range also covers Seattle.

Salary: $216,000–$270,000 USD base salary, plus potential equity and benefits.

Company

hirify.global develops AI data, evaluation, and full-stack technologies that help enterprises and governments build reliable AI systems.

What you will do

• Build observability, instrumentation, and monitoring capabilities for production agent behavior.
• Design scalable evaluation methodologies and metrics for agentic applications across customer use cases.
• Develop and own ML systems that detect drift, anomalies, and misalignment in production agents.
• Run rigorous experiments to validate model and agent performance improvements before release.
• Partner with software engineers to productionize research and experimental work.
• Collaborate with product, customer-facing engineering, data annotation, and other engineering teams to translate enterprise and government requirements into platform capabilities.

Requirements

• 5+ years of experience as an ML engineer or applied scientist working with production ML or LLM-powered systems.
• Strong experience in at least two areas: ML or agent evaluation infrastructure, agent system design, or new methods, reward models, and model training or fine-tuning.
• Hands-on experience with LLMs, tool use, planning, and multi-agent orchestration.
• Experience partnering with software engineers to bring research and experimental work into production.
• Rigorous statistical experimentation skills and the ability to work through ambiguous cross-functional requirements.
• Ability to provide and receive substantive technical feedback and mentor other engineers.

Nice to have

• Experience with RLHF, SFT, reward modeling, or verifiable-reward systems.
• Experience optimizing model or system latency, cost, or inference efficiency.
• Research publications, open-source contributions, or patents in agentic systems, LLMs, or applied ML.
• Experience in regulated or enterprise environments.
• Experience mentoring engineers or reviewing technical designs at a senior or staff level.

Culture & Benefits

• Work on AI systems used by leading enterprises and government organizations.
• Comprehensive health, dental, and vision coverage.
• Retirement benefits, a learning and development stipend, and generous paid time off.
• Potential equity compensation and, where applicable, a commuter stipend.
• Inclusive equal-opportunity workplace with reasonable accommodation support.