Senior AI Product Manager, Evals & Annotations

Location: United States (Remote). Offices are available in New York City and the San Francisco Bay Area for employees who wish to work in person; other employees receive a WFH monthly stipend.

Estimated annual salary: $200,000–$250,000, plus equity and benefits.

Company

Arize AI provides an AI and agent engineering observability and evaluation platform that helps teams build, test, monitor, and improve AI applications and agents.

What you will do

• Own the Evals & Annotations product area, including eval creation and execution, human review, annotation workflows, analysis, and iteration.
• Set product strategy and roadmap across customer needs, platform investments, UX improvements, and AI and agent development opportunities.
• Prototype evaluation workflows and partner with Design and Engineering to deliver intuitive, technically sound experiences.
• Work with Engineering on evaluation execution, backend systems, orchestration, data infrastructure, and technical tradeoffs.
• Conduct multiple customer conversations each week to understand evaluation workflows, identify problems, and validate product direction.
• Drive prioritization and products from problem definition through launch, adoption, and iteration while collaborating with founders and cross-functional teams.

Requirements

• Significant product management experience independently owning technically complex, developer-facing products; exceptional technical depth, product judgment, and ownership may compensate for a specific years-of-experience requirement.
• Strong understanding of LLM evaluations and AI development workflows, including automated evaluations, human feedback, experimentation, and iteration on AI applications and agents.
• Experience with developer tools, AI infrastructure, ML platforms, or adjacent technical products, plus a solid understanding of backend systems and software architecture.
• Strong technical depth and hands-on engineering ability, including independently debugging issues, navigating code, making small fixes or pull requests, building prototypes and proofs of concept, proposing implementations, and discussing tradeoffs with engineers.
• Strong product and UX instincts, with the ability to define and prototype core product experiences and collaborate with designers and design engineers.
• Strong ownership, initiative, bias toward action, and comfort with ambiguity, prioritization, and hands-on execution.
• Collaborative, low-ego approach with founders, engineers, and designers; able to engage in constructive debate and work across overlapping responsibilities.
• Experience working closely with technical customers, with strong communication skills across strategy and execution.

Nice to have

• Experience with LLM evaluation platforms, annotation systems, human-in-the-loop workflows, or agent orchestration.
• Backend or platform engineering experience, especially with distributed systems, workflow orchestration, or data infrastructure.
• Hands-on programming experience across backend and frontend technologies.
• Experience with LLM-as-a-judge, custom evaluators, experiment tracking, datasets, or feedback-driven improvement workflows.
• Previous engineering or substantial hands-on software development experience.
• Early-stage or high-growth startup experience, particularly in founder-led environments.

Culture & Benefits

• Remote-first work with a WFH monthly stipend for coworking spaces outside New York City and the San Francisco Bay Area.
• Medical, dental, vision, and 401(k) benefits.
• Unlimited paid time off and generous parental leave.
• Mental and wellness support, equity package, and a collaborative environment with direct founder and CPO involvement.
• Opportunities to engage with industry experts, researchers, and ethicists on responsible AI, alongside diversity and inclusion initiatives.