Senior AI QA Test Automation Engineer (AI)

Location: Limassol, Nicosia, Athens, Thessaloniki, or remote; hybrid workplace

Company

Engineering – Quality Assurance work focused on the CX Agent Suite and AI quality engineering.

What you will do

• Build scalable AI/ML QA frameworks, libraries, and tooling for the CX Agent Suite.
• Design evaluation pipelines for multi-agent responses, measuring accuracy, relevance, tone, hallucination rate, safety, guardrail compliance, and task completion.
• Evaluate and prototype Strands Agents or comparable orchestration frameworks for autonomous QA agents.
• Integrate AI-driven testing into GitLab CI/CD across development, test, staging, and production environments.
• Develop adaptive automation and quality analytics using OpenTelemetry, CloudWatch, X-Ray, and LangFuse evaluation data.
• Lead research, code reviews, workshops, and architectural guidance for QA and engineering teams.

Requirements

• BSc or MSc in Computer Science, AI, or a related discipline.
• 6+ years of hands-on QA or test automation experience designing and maintaining automation frameworks.
• 1+ year applying AI/ML to software testing or QA process improvement and bringing solutions into production workflows.
• Strong Python skills in AWS-centric environments; Java or TypeScript is advantageous.
• Experience with LLM-as-a-judge, human-in-the-loop evaluation, RAG, multi-agent orchestration, DeepEval or comparable frameworks, and agent-orchestration SDKs.
• Experience with GitLab CI/CD, Docker, Kubernetes/EKS, vector databases, MLOps pipelines, and enterprise AI tooling.

Nice to have

• Experience with autonomous QA agents, agentic orchestration, or self-evolving test suites.
• Experience with LangFuse, LangSmith, Arize, or other LLM observability tools.
• Knowledge of AI ethics, fairness, bias detection, gRPC, WebSockets, or HTTP/2.
• Experience with AWS Bedrock, GCP Vertex AI, or Azure AI.

Culture & Benefits

• Attractive remuneration package.
• Intellectually stimulating work environment.
• Continuous personal development opportunities.
• International training opportunities.

Hiring process

• Introductory chat with Talent Acquisition.
• First interview with the future team.
• Final interview.