Automation QA Engineer (AI)

TL;DR

Automation QA Engineer (Python/AI): Building and executing programmatic validation for a Retrieval-Augmented Generation pipeline supporting an industrial AI copilot, with an accent on retrieval quality, context precision, and grounding. Focus on designing RAGAS-based evaluation harnesses, integrating LLM quality gates into Azure DevOps, and detecting data ingestion, chunking, and retrieval regressions.

Location: Full remote work from Poland or Eastern Europe; physical location in Poland or Eastern Europe is required.

Company

hirify.global is a custom product engineering company serving multinational organizations and scaling startups through software engineering, consulting, analytics, and product development.

What you will do

• Build and run Python-based automated validation suites using RAGAS or equivalent LLM evaluation frameworks.
• Measure faithfulness, answer relevance, context precision, and other RAG pipeline quality metrics.
• Integrate automated evaluation scorecards, regression triggers, and quality gates into Azure DevOps pipelines.
• Validate documentation ingestion, chunking consistency, data quality, and vector-store retrieval accuracy.
• Produce programmatic scorecards and regression reports for project phase-gate approvals.
• Collaborate with ML implementation engineers to reproduce and resolve chunking and retrieval issues.

Requirements

• 3+ years of experience in backend test automation, data quality engineering, or Python software development.
• Strong Python skills for backend testing, scripting, and data manipulation.
• Hands-on experience or deep technical familiarity with RAGAS or equivalent automated LLM validation engines.
• Practical experience with CI/CD pipelines, automated regression triggers, and quality gates, preferably in Azure DevOps.
• Basic understanding of PostgreSQL, data profiling, and search indexing concepts.
• Must be physically located in Poland or Eastern Europe.

Culture & Benefits

• Full remote working possibilities with collaboration in a cross-functional delivery squad.
• Access to meetups, conferences, workshops, Udemy, language courses, and company-paid certifications.
• Opportunities for internal mobility across diverse technology domains.
• Company-paid premium medical coverage through Luxmed.
• MyBenefit cafeteria platform with flexible perks.
• Collaborative, open-door culture focused on innovation and professional growth.

Hiring process

• Submit an application for consideration.