Назад
2 месяца назад

Product Manager, LLM Evaluation & User Insight (E-Commerce Recommendations)

141 588 - 234 080$
Формат работы
onsite
Тип работы
fulltime
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Product Manager, LLM Evaluation & User Insight (E-Commerce Recommendations) (LLM evaluation/e-commerce recommendations): Designing comprehensive automated, model-based, and human evaluation systems for LLM-powered product discovery with an accent on benchmark datasets, quality metrics, and user research. Focus on preference modeling, red-teaming, failure-mode analysis, and translating model insights into improvements for recommendation, search, and AI discovery systems.

Location: Seattle, United States; fully in-person schedule up to 5 days a week

Salary: $141,588–$234,080 annually, plus potential bonuses, incentives, and restricted stock units.

Company

TikTok USDS Joint Venture operates TikTok-related applications with a focus on U.S. user data protection, privacy, cybersecurity, trust, and safety.

What you will do

  • Design and own automated, model-based, and human evaluation methodologies for LLM capabilities in e-commerce.
  • Define quality metrics and build standardized benchmark datasets.
  • Conduct quantitative and qualitative research into buyer and creator needs, mental models, and pain points.
  • Lead human annotation, preference modeling, pairwise comparisons, RLHF data strategies, and red-teaming initiatives.
  • Translate LLM quality insights into actionable improvements for Recommendation, Search, and AI Discovery product and engineering teams.
  • Diagnose model performance and failure modes while recommending improvements to multi-objective ranking and context-aware delivery.

Requirements

  • Bachelor’s degree or higher.
  • At least 1 year of experience in Product Management, Data Product Management, or AI Quality/Evaluation.
  • Hands-on experience or deep understanding of LLM evaluation methodologies and prompt engineering.
  • Strong quantitative and qualitative analysis, metric design, and ability to translate research and data insights into technical specifications.
  • Ability to align cross-functional stakeholders, including product managers, ML engineers, data scientists, and operations teams.
  • Ability to work fully in person in Seattle, up to 5 days per week.

Nice to have

  • 3+ years of experience in product management focused on LLM or GenAI evaluation, AI quality, user research, data annotation strategy, or machine learning applications.
  • Experience in e-commerce, search and recommendation, or conversational AI.
  • Familiarity with LLM training and fine-tuning, RLHF, and AI safety procedures.

Culture & Benefits

  • Medical, dental, and vision insurance from day one.
  • 401(k) savings plan with company match, paid parental leave, disability coverage, and life insurance.
  • 10 paid holidays, 10 paid sick days, and 17 days of paid personal time.
  • Work environment centered on curiosity, humility, resilience, continuous learning, inclusion, and collaboration.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →