Senior QA Automation Engineer (AI)

TL;DR

Senior QA Automation Engineer (AI): Designing and executing testing scenarios for conversational agents and generative AI systems with an accent on LLM validation and data consistency. Focus on identifying hallucinations, bias, and toxicity while ensuring the reliability of non-deterministic AI outputs.

Location: Must be based in Spain (Hybrid)

Company

Boutique technology services and consulting firm specializing in digital transformation and AI enablement for global organizations.

What you will do

• Design and execute GenAI testing scenarios for chatbot interactions, intent recognition, and edge-case behavior.
• Validate LLM outputs using prompt and grounding validation, golden sets, and non-deterministic assertions.
• Perform safety and quality checks, including testing for hallucination, bias, toxicity, and PII leakage.
• Validate structured and unstructured data outputs to ensure consistency, accuracy, and compliance.
• Collaborate with ML and NLP teams to define comprehensive evaluation metrics for agent performance.
• Integrate AI-driven tools like Copilot or Claude into the QA lifecycle to accelerate test design and defect analysis.

Requirements

• 8+ years of experience in QA engineering with proven expertise in automation.
• 5+ years of experience in testing complex data systems.
• Strong proficiency in automation testing using JS and Python.
• Experience with manual and automated API testing.
• Arabic language proficiency.
• Must be based in Spain.

Nice to have

• Familiarity with chatbot frameworks, LLMs, or conversational testing.
• Experience using LLM-based assistants (e.g., GitHub Copilot, ChatGPT) in test execution.
• Exposure to QA automation tools such as PyTest, Postman, or dbt tests.

Culture & Benefits

• Highly competitive compensation package.
• Flexible and hybrid working environment.
• Medical insurance.
• Opportunity to work within a multicultural community of tech experts in a fast-growing organization.