AI QA Engineer

TL;DR

AI QA Engineer (AI/LLM): Testing AI-powered web and mobile applications by defining correct outputs, designing probabilistic evaluation criteria, and investigating system behavior with an accent on hallucinations, retrieval quality, business rules, and access control. Focus on building test plans for non-deterministic systems, diagnosing failure modes, and validating API data flows with developers, product managers, and AI engineers.

Location: Remote from Ukraine

Company

Remote-first product design, engineering, and marketing studio building AI-powered web and mobile applications for major brands, startups, and research organizations.

What you will do

• Define acceptance criteria and determine what correct output means for AI-powered features with probabilistic results.
• Design test plans, scenarios, checklists, and evaluation approaches for AI systems.
• Perform manual functional, regression, and exploratory testing across web and mobile applications.
• Evaluate AI outputs for hallucinations, incomplete retrieval, confidently incorrect answers, prompt sensitivity, and business-rule violations.
• Inspect REST API requests and responses using browser DevTools and Postman to verify underlying data flows.
• Document reproducible defects and communicate findings to technical and non-technical stakeholders.

Requirements

• 1.5+ years of experience in QA or software engineering.
• Strong understanding of QA processes, methodologies, terminology, test plans, test cases, SDLC, and regression testing.
• Experience manually testing web and mobile applications.
• Practical experience with REST APIs, Postman, or browser DevTools.
• Upper-Intermediate English required for daily communication with international clients.
• Ability to form hypotheses, resolve ambiguity, define acceptance criteria, and investigate AI failure modes.

Nice to have

• Basic SQL or experience querying data directly.
• Familiarity with LLMs, RAG pipelines, or AI chatbots.
• Experience testing role-based access control.

Culture & Benefits

• 20 days of paid vacation during the first three years, increasing to 25 days later.
• Paid sick leave, 10 national holidays, and two company days off.
• Well-being budget and health insurance allowance.
• Maternity and paternity leave, accounting assistance, and hardware based on business needs.
• Professional development course and certification reimbursement of up to 100% with manager approval.
• Collaborative engineering culture with a closely connected team.

Hiring process

• HR interview.
• Technical interview with the QA Team Lead and QA Manual Engineer.
• Offer.