Вакансия из Hirify Global, списка международных tech-компаний Для мэтча и отклика нужен Plus
Пожаловаться
Зарплата и рынок
ЗП не указанаразвернуть
На похожих ролях по рынку
$175к$203кмедиана$245к
49 в выборке
На СНГ рынке: $4.3к/мес ($3.6к - $5к)
76
Хорошая вакансия
развернуть
Работа по оценке AI-систем описана чётко и связана с актуальной технической областью, но занятость на 50% и короткий проектный период могут ограничить непрерывность работы и долгосрочную ответственность.
Неполная загрузка
Актуальные технологии
Чёткие задачи
?Клиентские проекты
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Подключить Plus
Описание вакансии
Текст:
Hirify AI/Оригинал
TL;DR
Senior AI Quality & Evaluation Engineer (Python/RAG/AWS): Building independent evaluation harnesses for retrieval quality, authorization boundaries, content freshness, and model or configuration changes with an accent on golden question suites, negative entitlement tests, and regression monitoring. Focus on automating CI-scheduled tests, scoring relevance and grounding, detecting drift, and reporting root causes across content, chunking, ranking, and permissions.
Location: USA; remote
Company
hirify.global is an AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. Following the acquisition of Tooploox, hirify.global provides end-to-end delivery from advisory and solution design through custom AI development and enterprise implementation.
What you will do
Build and version golden question suites with expected sources, passages, and thresholds for relevance, grounding, and citation accuracy.
Design negative entitlement tests for unentitled identities and raise restricted-content failures as severity-one alarms.
Create freshness and retirement tests for indexing lag and removal of retired content after synchronization.
Automate scheduled and change-triggered evaluation runs, publishing scores, trends, and regression alarms to the quality dashboard.
Define QA gates for remediated content, run evaluations before pilot changes in weeks 7–9, and report root causes to the appropriate owners.
Write the evaluation runbook and train client stewards and platform owners to maintain suites and interpret regressions.
Requirements
3+ years of software quality engineering or test automation experience with Python, including CI-based scheduled test harnesses.
Hands-on production evaluation of search or RAG systems, including ground-truth sets, relevance and grounding scoring, and drift tracking.
Experience designing security-focused negative tests for authorization, entitlements, and data exposure, with severity reporting suitable for client security teams.
Ability to work with AWS, including Bedrock and OpenSearch APIs, CloudWatch, and scheduled jobs deployed with infrastructure code.
Clear written reporting for client stewards and engineers.
Nice to have
AWS Certified AI Practitioner or Machine Learning Engineer Associate certification.
Experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration.
Contact center or knowledge management domain experience.
Familiarity with SOC 2 or ISO 27001 evidence practices.
Culture & Benefits
Full-time role with a 50% allocation across weeks 3–10.
Work with clients across startup innovation and enterprise transformation projects.
Global collaboration across continents and cultures in an inclusive environment.
Emphasis on continuous learning, innovation, and ethical AI standards.
Reports to the POD Lead and works day-to-day with the retrieval engineer.
Будьте осторожны
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →