12 дней назад
Engagement Manager, Agentic AI Workflow Evaluations (AI)
75 - 85$
Вакансия из Hirify Global, списка международных tech-компанийДля мэтча и отклика нужен Plus
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Описание вакансии
Текст:
TL;DR
Engagement Manager, Agentic AI Workflow Evaluations (AI) (AI evaluation and delivery leadership): Leading an onsite team that evaluates complex agentic AI workflows for safety, user intent, consent, and scoring quality with an accent on customer delivery, rubric auditing, and escalation management. Focus on managing eleven reviewers and a QA lead, calibrating evaluation standards, defending scoring decisions, and expanding enterprise AI evaluation engagements.
Location: In office in San Jose, California, United States
Expected salary: $75–85 per hour, based on experience, skills, and qualifications.
Company
is a global data engineering company providing data, evaluation frameworks, human expertise, and services for generative AI and AI solutions.
What you will do
- Own end-to-end delivery for an onsite team evaluating complex, real-world agentic AI workflows.
- Manage and develop eleven employees, including hiring, onboarding, performance management, and retention.
- Act as the primary onsite contact for customer program and technical leads, reporting delivery metrics and resolving issues.
- Audit reviewer trajectories, rubric application, and scoring rationales for quality and defensibility.
- Partner with the QA lead on calibration, quality metrics, evaluation, and rubric refinement.
- Manage safety-related escalations, staffing and cost forecasts, information security, privacy, and facility access.
Requirements
- Bachelor's degree or equivalent practical experience.
- 5+ years of professional experience, including direct management of a delivery team of 10–20 people.
- Experience owning customer-facing services or delivery engagements for enterprise or technology clients.
- Significant experience with AI/ML evaluation, annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review.
- Ability to critically assess rubric-scored work and independently determine whether scores are correct.
- Strong written communication skills for technical reporting and escalation memos.
Culture & Benefits
- Dedicated onsite team operating in isolated customer test environments.
- Work focused on responsible advancement of artificial intelligence and evaluation of AI systems at scale.
- Role includes direct customer interaction and collaboration with delivery, QA, technical, and commercial teams.
- Facility access, information security, and privacy practices are required in the customer environment.
Будьте осторожны: если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого - это мошенники. Обязательно жмите "Пожаловаться" или пишите в поддержку. Подробнее в гайде →