Senior QA Automation Engineer (AI)

TL;DR

Senior QA Automation Engineer (AI): Designing and executing quality assurance strategies for complex data flows and AI-driven conversational agents with an accent on LLM validation and generative AI safety. Focus on building non-deterministic assertions, validating prompt grounding, and preventing hallucinations or bias in AI outputs.

Location: Hybrid in Dubai (2-3 days at the office per week)

Company

A boutique technology services and consulting firm helping global organisations solve complex business challenges through digital transformation and AI enablement.

What you will do

• Design and execute generative AI testing scenarios for chatbot interactions, intent recognition, and edge-case behavior.
• Develop LLM test designs, including prompt and grounding validation, golden sets, and non-deterministic assertions.
• Perform safety and quality checks to identify hallucinations, bias, toxicity, and PII leakage.
• Validate structured and unstructured data outputs, ensuring consistency, accuracy, and compliance.
• Collaborate with ML and NLP teams to define comprehensive evaluation metrics and criteria for agent performance.
• Integrate AI-driven tools like Copilot or Claude into the QA lifecycle to accelerate test design and defect analysis.

Requirements

• Fluent Arabic language skills (spoken and written).
• 8+ years of experience in QA engineering with proven expertise in automation.
• 5+ years of experience testing data systems.
• Strong proficiency in automation testing using JavaScript and Python.
• Experience with API testing, both manual and automated.
• Familiarity with chatbot frameworks, LLMs, and conversational testing.

Nice to have

• Exposure to QA automation tools such as PyTest, Postman, or dbt tests.
• Experience using LLM-based assistants (e.g., GitHub Copilot, ChatGPT) in test execution.

Culture & Benefits

• Opportunity to join a fast-growing organization with triple-digit growth.
• Part of a multicultural community of senior tech experts.
• Highly competitive compensation package.
• Flexible and hybrid working environment.
• Medical insurance.