Product Manager, LLM Evaluation & User Insight (E-Commerce Recommendations)

Location: San Jose, United States; fully in-person schedule up to 5 days a week

Salary: $149,040–$246,400 annually

Company

hirify.global operates a U.S. content ecosystem focused on data privacy, cybersecurity, trust and safety, and protecting U.S. user data.

What you will do

• Design and own automated, model-based, and human evaluation methodologies for LLMs in e-commerce contexts.
• Define quality metrics and build standardized benchmark datasets.
• Conduct quantitative and qualitative research into buyer and creator needs, mental models, and pain points.
• Lead human annotation, preference modeling, pairwise comparisons, and red-teaming initiatives.
• Connect LLM quality insights with Recommendation, Search, and AI Discovery product and engineering teams.
• Diagnose model performance and failure modes and recommend improvements to ranking and context-aware delivery logic.

Requirements

• Bachelor’s degree or higher and at least 1 year of experience in Product Management, Data Product Management, or AI Quality/Evaluation.
• Hands-on experience or deep understanding of LLM evaluation methodologies and prompt engineering.
• Strong quantitative and qualitative analysis, metric design, and ability to translate research and data insights into technical specifications.
• Ability to align cross-functional stakeholders, including product managers, machine learning engineers, data scientists, and operations teams.
• Preferred experience includes 3+ years in LLM/GenAI evaluation, AI quality, user research, data annotation strategy, or machine learning applications.
• Technical familiarity with LLM training and fine-tuning, RLHF, and AI safety procedures is preferred.

Nice to have

• Experience in e-commerce, search and recommendation systems, or conversational AI.
• Experience with LLM training, fine-tuning, RLHF, and AI safety standard operating procedures.

Culture & Benefits

• Medical, dental, and vision insurance from day one.
• 401(k) savings plan with company match, life insurance, disability coverage, and wellbeing benefits.
• Paid parental leave, 10 paid holidays, 10 paid sick days, and 17 days of paid personal time.
• Inclusive workplace focused on creativity, curiosity, continuous learning, and collaboration.
• On-site work supports rapid alignment, real-time decision-making, and integrated execution.