Senior AI Test Engineer (AI)

TL;DR

Senior AI Test Engineer (AI): Designing and implementing comprehensive testing strategies for LLM-based applications with an accent on RAG pipelines, output accuracy, and ethical compliance. Focus on building automated testing solutions, analyzing edge cases, and conducting performance and security testing within a cloud-based AI environment.

Location: Remote (Must be based in Poland)

Company

hirify.global is a forward-thinking IT services company that partners with global technology leaders to deliver innovative solutions through a diverse, international team.

What you will do

• Design and implement testing strategies tailored for LLM-based applications, including RAG pipelines.
• Develop and maintain automated testing frameworks for continuous evaluation of AI systems.
• Conduct edge case, bias, and fairness testing to ensure model reliability and ethical compliance.
• Collaborate on security testing to identify vulnerabilities such as prompt injection and data leakage.
• Perform performance testing including response time analysis, load testing, and resource monitoring.
• Maintain comprehensive test documentation and risk analysis specific to AI implementations.

Requirements

• 5+ years of experience in software testing with at least 1 year focused on GenAI applications.
• Strong proficiency in Python and testing frameworks like pytest.
• Knowledge of LLM architectures, RAG systems, and common AI failure modes.
• Experience with test automation, CI/CD pipelines (GitLab/GitHub Actions), and cloud platforms (AWS).
• Understanding of monitoring tools (Grafana) and security testing methodologies.
• Degree in Computer Science, Software Engineering, or equivalent field.

Culture & Benefits

• Comprehensive benefits including private medical care, Multisport card, and veterinary package.
• Access to professional development via Udemy for Business and partnerships with Microsoft, AWS, and Snowflake.
• Opportunity to work in an innovative environment with a diverse, international team.
• Engagement in community initiatives, hackathons, and company-wide team-building events.
• Welcome pack provided upon starting.