Senior QA Automation Engineer (AI & LLM Testing)

TL;DR

Senior QA Automation Engineer (AI & LLM Testing): Designing and maintaining automated test suites for web applications and APIs with an accent on LLM quality evaluation and hallucination risk mitigation. Focus on building end-to-end UI automation with Playwright and creating specialized prompt tests using Promptfoo.

Location: Remote, Hungary

Company

Digital product engineering company specializing in building scalable products and services for global clients across various industries.

What you will do

• Design and maintain automated test suites for web applications, APIs, and AI-powered features.
• Evaluate LLM outputs for response accuracy, relevance, consistency, and hallucination risks.
• Create prompt tests and quality checks using Promptfoo or similar evaluation frameworks.
• Build end-to-end UI automation using Playwright following page object models.
• Develop reusable Python scripts for API testing and LLM quality evaluation.
• Collaborate with product managers and developers to define test strategies and communicate risks.

Requirements

• Strong experience in QA Automation and software testing.
• Hands-on experience testing AI/LLM-based features, including prompt testing and output validation.
• Proficiency in Playwright for end-to-end UI automation.
• Strong Python scripting skills for automation and custom quality checks.
• Experience with Postman and REST API testing with CI/CD integration.
• Must be based in Hungary.

Nice to have

• Experience with mobile test automation using Appium or XCUITest.
• Understanding of iOS and Android platform-specific testing.
• Experience integrating automated tests into CI/CD pipelines.
• Familiarity with MCP or agent-assisted testing in Playwright.

Culture & Benefits

• Dynamic and non-hierarchical work culture.
• Opportunity to work on a global scale across 39 countries.
• Collaborative environment focused on mentorship and solving complex client challenges.
• Work with cutting-edge AI coding assistants like GitHub Copilot.