Назад
Company hidden
8 дней назад

Senior AI Quality & Evaluation Engineer

Формат работы
remote (только USA)
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Senior AI Quality & Evaluation Engineer (Python/RAG/AWS): Building independent evaluation harnesses for retrieval quality, authorization boundaries, content freshness, and model or configuration changes with an accent on golden question suites, negative entitlement tests, and regression monitoring. Focus on automating CI-scheduled tests, scoring relevance and grounding, detecting drift, and reporting root causes across content, chunking, ranking, and permissions.

Location: USA; remote

Company

hirify.global is an AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. Following the acquisition of Tooploox, hirify.global provides end-to-end delivery from advisory and solution design through custom AI development and enterprise implementation.

What you will do

  • Build and version golden question suites with expected sources, passages, and thresholds for relevance, grounding, and citation accuracy.
  • Design negative entitlement tests for unentitled identities and raise restricted-content failures as severity-one alarms.
  • Create freshness and retirement tests for indexing lag and removal of retired content after synchronization.
  • Automate scheduled and change-triggered evaluation runs, publishing scores, trends, and regression alarms to the quality dashboard.
  • Define QA gates for remediated content, run evaluations before pilot changes in weeks 7–9, and report root causes to the appropriate owners.
  • Write the evaluation runbook and train client stewards and platform owners to maintain suites and interpret regressions.

Requirements

  • 3+ years of software quality engineering or test automation experience with Python, including CI-based scheduled test harnesses.
  • Hands-on production evaluation of search or RAG systems, including ground-truth sets, relevance and grounding scoring, and drift tracking.
  • Experience designing security-focused negative tests for authorization, entitlements, and data exposure, with severity reporting suitable for client security teams.
  • Ability to work with AWS, including Bedrock and OpenSearch APIs, CloudWatch, and scheduled jobs deployed with infrastructure code.
  • Clear written reporting for client stewards and engineers.

Nice to have

  • AWS Certified AI Practitioner or Machine Learning Engineer Associate certification.
  • Experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration.
  • Contact center or knowledge management domain experience.
  • Familiarity with SOC 2 or ISO 27001 evidence practices.

Culture & Benefits

  • Full-time role with a 50% allocation across weeks 3–10.
  • Work with clients across startup innovation and enterprise transformation projects.
  • Global collaboration across continents and cultures in an inclusive environment.
  • Emphasis on continuous learning, innovation, and ethical AI standards.
  • Reports to the POD Lead and works day-to-day with the retrieval engineer.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →