Mid-Level AI & Automation QA Engineer (AI-Driven Testing)

Location: Porto, Portugal; hybrid work model. Candidates must have an existing legal right to work in the European Union.

Company

hirify.global provides a client opportunity focused on testing AI-powered features across conversational AI, productivity tools, and rating systems.

What you will do

• Develop and maintain automated test frameworks with Playwright, Cypress, Selenium, or similar tools.
• Create Python or JavaScript/TypeScript testing solutions for automation and evaluation pipelines.
• Design evaluation datasets and automated pipelines, including LLM-based judgment approaches.
• Analyze AI outputs for hallucinations, errors, bias, fairness, and compliance issues.
• Integrate AI evaluation and functional tests into CI/CD pipelines.
• Test conversational AI dialogue flows, edge cases, and adversarial prompts while using observability and tracing tools to debug production model behavior.

Requirements

• 3+ years of experience in QA or SDET roles, particularly automation testing.
• Hands-on experience with Playwright, Cypress, Selenium, or similar automated testing frameworks.
• Proficiency in Python or JavaScript/TypeScript.
• Knowledge of AI testing challenges, including prompt regression, hallucination detection, output scoring, and evaluation datasets.
• Experience with CI/CD pipelines, AI observability or tracing tools, and root-cause analysis of code and probabilistic AI outputs.
• Fluent English and an existing legal right to work in the European Union are required.

Nice to have

• Experience with AI-assisted test generation or self-healing scripts.
• Knowledge of LLM evaluation methods and AI debugging tools.
• QA or AI safety certifications.

Culture & Benefits

• Hybrid work in Porto.
• Work on AI reliability, safety, and regression prevention for cutting-edge AI applications.
• Professional growth in a dynamic environment focused on advancing AI testing.