Назад
Company hidden
6 дней назад

AI Evaluation Specialist (AI)

155 000$
Формат работы
hybrid
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
AI Evaluation Specialist (AI/Generative AI): Evaluating enterprise AI tools, models, agents, and AI-enabled applications with an accent on quality, accuracy, safety, privacy, security, usability, and business readiness. Focus on designing repeatable evaluation frameworks, testing RAG and tool-enabled workflows, comparing vendors, and analyzing model behavior and failure modes.

Location: Atlanta, Georgia, United States; hybrid work model

Salary: $155,000

Company

hirify.global delivers outsourced services and workforce solutions across North America, supporting clients with efficient, quality-focused delivery.

What you will do

  • Plan and execute hands-on evaluations of enterprise generative AI tools, models, agents, and AI-enabled applications using representative business scenarios, workflows, prompts, datasets, and user personas.
  • Assess response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and task effectiveness.
  • Design and maintain evaluation frameworks, scorecards, rubrics, benchmarks, and measurable acceptance criteria.
  • Evaluate safety, privacy, security, data handling, governance, responsible AI, and compliance considerations with relevant stakeholders.
  • Compare AI products, models, configurations, and vendors, including functionality, integrations, performance, cost, and proof-of-concept results.
  • Evaluate RAG, enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol-enabled workflows while sharing findings and recommendations.

Requirements

  • 5+ years of experience in model evaluation or model quality assurance, including hands-on evaluation of AI-enabled products or services.
  • Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
  • Hands-on experience with generative AI tools and large language models, including prompt design, systematic testing, output analysis, hallucination identification, and analysis of other failure modes.
  • Experience assessing AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost.
  • Familiarity with large language models, retrieval-augmented generation, AI agents, APIs, enterprise search, and tool or connector integrations.
  • Bachelor’s degree.
  • Must be a U.S. Citizen or Lawful Permanent Resident (Green Card Holder).
  • Must be able to obtain Public Trust clearance.

Nice to have

  • Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models such as Rosalind.
  • Experience with automated LLM evaluation frameworks, model-as-a-judge methods, human-in-the-loop evaluation, red teaming, or adversarial testing.
  • Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows.
  • Experience developing dashboards or reporting mechanisms for evaluation metrics, quality trends, risk findings, and regression results.
  • Ability to analyze datasets and evaluation results using Python, SQL, spreadsheets, notebooks, or analytics platforms.
  • Industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI.
  • Master’s degree preferred.

Culture & Benefits

  • Direct-hire position with a hybrid work model.
  • Collaborative environment involving product, data science, engineering, security, privacy, legal, governance, and business stakeholders.
  • Eligible employees may receive medical, dental, vision, spending account, life insurance, voluntary plan, and 401(k) benefits.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →