AI Engineer (Quality)

TL;DR

AI Engineer (Quality): Designing and building a unified evaluation platform and observability systems for AI agents in complex audit workflows with an accent on model reliability, production feedback loops, and automated pipelines. Focus on orchestrating LLMs, implementing rigorous evaluation harnesses, and ensuring AI agents meet enterprise-grade standards.

Location: Must be based in the US; role involves hybrid collaboration at the San Francisco, CA office.

Salary: $170K – $220K

Company

hirify.global is an early-stage startup building AI agents and software to automate and streamline mission-critical audit and assurance workflows for global commerce and capital markets.

What you will do

• Design and build a unified evaluation platform serving as the single source of truth for agentic systems.
• Develop observability systems that trace execution, failure modes, and production feedback loops.
• Create automated pipelines to evaluate new models against critical workflows within hours of release.
• Orchestrate LLMs, tools, and retrieval systems into reliable, cohesive agent experiences.
• Establish evaluation standards, best practices, and processes across the engineering organization.
• Partner with product and ML engineers to integrate evaluation requirements from the early development stages.

Requirements

• Must be authorized to work in the United States and able to work from the San Francisco, CA office.
• Proven experience shipping production software in complex, real-world systems.
• Proficiency in TypeScript, React, Python, and Postgres.
• Experience deploying LLM-powered features and building observability or tracing infrastructure for AI systems.
• Hands-on experience with evaluation platforms like LangSmith or Langfuse.
• Comfort operating in ambiguity and a strong desire to take ownership of large product areas.

Nice to have

• Experience with RAG architectures, vector databases, and embedding models.
• Deep empathy for professional-grade mission-critical software.

Culture & Benefits

• Competitive compensation with meaningful equity ownership.
• Flexible work schedules and flexible PTO policy.
• 401k retirement plan.
• Wellness benefits including a bundle of free therapy sessions.
• Technology and Work-from-Home reimbursement.
• Supportive, inclusive, and mission-driven team culture.