Automation QA Engineer (AI)

TL;DR

Automation QA Engineer (AI): Own the evaluation lifecycle, offline acceptance testing, and KPI measurement for the AS Copilot RAG pipeline with an accent on reducing AI hallucinations in code generation and optimizing retrieval latency. Focus on building golden datasets, implementing RAGAS evaluation harness, automated CI/CD regression testing, and tracking hallucinations with root-cause taxonomies.

Location: PL-Remote (Poland)

Company

Custom product engineering company supporting multinational organizations and scaling startups with a global team of over 4,000 professionals and delivery centers in Wrocław and Gdańsk.

What you will do

• Own evaluation lifecycle, offline acceptance testing, and KPI measurement for the AS Copilot RAG pipeline
• Lead co-creation and management of the project's golden dataset to benchmark AI performance
• Implement and manage RAGAS evaluation harness and automated CI/CD regression testing
• Track, classify, and build root-cause taxonomies for LLM hallucinations, focusing on code-generation correctness
• Build synthetic test sets and establish baseline metrics for Faithfulness, Context Precision, and Answer Relevance
• Measure and monitor pipeline latency, validating P95 targets under concurrent load

Requirements

• Mid-to-Senior level experience in Data Science, Machine Learning Evaluation, AI Quality Assurance, or Data Engineering
• Deep hands-on experience with LLM evaluation frameworks (e.g., RAGAS, DeepEval, TruLens) and benchmarks
• Strong proficiency in Python, CI/CD tools (especially Azure DevOps), and integrating test suites into pipelines
• Experience with databases (PostgreSQL) and integrating telemetry/observability (e.g., Azure App Insights)
• Strong analytical mindset for error analysis, taxonomies, and identifying embedding drift
• Highly collaborative and data-driven, comfortable working with client SMEs

Culture & Benefits

• Strong community with top professionals in a friendly, open-door environment
• Growth focus through large-scale projects, internal events, Udemy, language courses, and certifications
• Endless opportunities via internal mobility and diverse domains
• Flexibility with full remote working possibilities
• Company-paid medical insurance, mental health support, financial & legal consultations