AI QA Engineer

Location: Europe and Ukraine; remote-first with flexible remote and hybrid working options

Company

hirify.global is a global digital services and software solutions company working across supply chain, healthtech, collaboration technology, and gaming.

What you will do

• Design and execute quality strategies for AI-powered warehouse assistants, copilots, and autonomous agents.
• Build LLM evaluation frameworks, golden datasets, benchmark scenarios, and agent trajectory tests for warehouse workflows.
• Test hallucinations, factuality, grounding, response quality, RAG retrieval accuracy, prompt injection, jailbreaks, model drift, and output degradation.
• Create scalable functional, API, integration, and end-to-end automation frameworks using suitable testing tools.
• Validate integrations across WMS, ERP, TMS, OMS, WCS, and third-party platforms, while implementing CI/CD quality gates and regression suites.
• Define AI quality standards, KPIs, audit-ready evidence, and compliance practices while collaborating with product, architecture, UX, and engineering teams.

Requirements

• 5+ years of experience in quality assurance and test automation.
• 2+ years of experience testing AI-enabled applications, LLM solutions, or agentic systems.
• Strong Python skills and experience with Playwright, Selenium, Cypress, or similar automation frameworks.
• Experience with API, integration, database, CI/CD, DevOps, and cloud-native testing, including Azure or AWS.
• Knowledge of AI evaluation frameworks, AI-specific failure modes, MCP architecture, and agent-to-tool integrations.
• Understanding of software testing methodologies, test design, risk-based testing, and enterprise quality practices.

Nice to have

• Experience with supply chain, logistics, WMS, TMS, OMS, ERP, warehouse operations, or fulfillment processes.
• Experience with LangSmith, Braintrust, MLflow, Arize, or Evidently AI.
• Familiarity with the EU AI Act, ISO 42001, and responsible AI governance frameworks.
• ISTQB CT-AI certification, AI testing credentials, or contributions to AI testing frameworks and open-source projects.
• Leadership experience driving AI quality strategy across multiple teams.

Culture & Benefits

• Flexible hours and remote-first work with the option to work from home or an office.
• Hardware and software setup required for work.
• Health insurance, vacation, sick leave, holidays, and paid maternity and paternity leave.
• Learning and development resources, mentorship, individual development planning, online courses, and certification preparation.
• Transparent communication, virtual team-building activities, company events, and social programs.