Senior Software Engineer, Agent Oversight (AI)

TL;DR

Senior Software Engineer, Agent Oversight (AI): Building platform infrastructure, observability tooling, evaluation harnesses, APIs, and data pipelines for production agentic AI applications with an accent on reliability, scalability, and evaluation quality. Focus on designing telemetry and improvement loops, solving distributed-systems challenges, and turning ML evaluation workflows into dependable platform capabilities for enterprise and government customers.

Location: Onsite in San Francisco, CA or New York, NY. The salary range also references Seattle for eligible roles.

Salary: $216,000–$270,000 USD base salary, plus potential equity and benefits.

Company

hirify.global develops AI data infrastructure and full-stack technologies that help enterprises and governments build, deploy, evaluate, and oversee reliable AI systems.

What you will do

• Design and build platform capabilities for deploying, monitoring, and evaluating production agentic applications.
• Develop reliable APIs and data pipelines for agent telemetry, evaluation signals, and performance metrics.
• Partner with ML engineers and applied researchers to productionize evaluation and model-improvement workflows.
• Own the reliability, scalability, and observability of platform components serving enterprise and government customers.
• Collaborate with product, forward-deployed engineering, and customers to translate deployment needs into platform features.
• Deliver features end-to-end through system design, implementation, debugging, testing, and experimentation.

Requirements

• 4+ years of professional software engineering experience with backend and distributed systems, APIs, and data pipeline design.
• Production experience with ML/LLM-powered products or platforms, including evaluation, observability, experimentation, agent runtimes, model-serving-adjacent services, or telemetry pipelines.
• Working knowledge of LLM and ML production behavior, including evaluation signals, failure modes, prompt and tool-calling workflows, data quality, and offline-versus-live evaluation tradeoffs.
• Experience building infrastructure or internal platforms used by other engineering teams and owning components from design through production.
• Ability to work independently in an ambiguous, fast-changing environment and collaborate closely with ML engineers, researchers, customers, and product teams.
• Strong communication, design and code review, feedback, problem-solving, and mentoring skills.

Nice to have

• Deep experience with production observability, monitoring, or evaluation systems for ML/LLM products.
• Familiarity with agent architectures, tool use, planning, and multi-agent orchestration.
• Exposure to MLOps, feature stores, model serving, or experiment infrastructure.
• Experience in regulated or enterprise environments and at senior or staff-level design review or mentoring.

Culture & Benefits

• Work on AI infrastructure for commercial and public-sector use cases.
• Comprehensive health, dental, and vision coverage.
• Retirement benefits, learning and development stipend, and generous paid time off.
• Potential commuter stipend and equity compensation, subject to eligibility and approval.
• Inclusive equal-opportunity workplace with reasonable accommodation support.

Hiring process

• Salary placement is determined during the interview process based on location, skills, experience, qualifications, interview performance, and education or training.
• Candidates must wait 90 days before being reconsidered for the same role.