Senior AI Test Automation Engineer (AI)

TL;DR

Senior AI Test Automation Engineer (Python/AI): Designing and implementing technical test automation solutions for an AI and MLOps platform with an accent on RAG systems, LLM-based assistants, and agentic workflows. Focus on building scalable automation frameworks, evaluation pipelines for LLM outputs, and integrating tests into GitLab CI/CD pipelines.

Location: Madrid, Spain

Company

hirify.global is a global wealth management firm focusing on creating value beyond wealth through innovative financial services.

What you will do

• Define and evolve the technical test automation approach and framework architecture for the ML & AI ART.
• Design reusable and scalable test automation patterns using Playwright-Python, pytest, and BDD frameworks (Behave/pytest-bdd).
• Implement AI/ML-specific test cases and evaluation pipelines for LLM outputs.
• Integrate and orchestrate automated tests within GitLab CI/CD pipelines using Docker and Kubernetes.
• Collaborate with Product Owners, ML/MLOps Engineers, and Data Engineers within a SAFe Agile environment.
• Manage test data strategy, ensuring the use of synthetic or anonymized data to meet confidentiality expectations.

Requirements

• 5–7 years of experience in test automation with substantial hands-on Python expertise.
• Proven ability to design and own test automation frameworks and maintainability patterns.
• Experience integrating automated tests into CI/CD pipelines (GitLab or equivalent).
• Proficiency with Playwright-Python, requests, and Gherkin/BDD tools.
• Fluency in English is required; Spanish is a plus.
• Must be based in Madrid, Spain.

Nice to have

• Experience testing AI/ML systems, including evaluation of non-deterministic LLM responses and RAG workflows.
• Experience working in regulated environments such as financial services, healthcare, or pharma.
• ISTQB certification (Foundation level) or SAFe (SP, SSM) and DevOps certifications.

Culture & Benefits

• Environment that values individual qualities, entrepreneurship, and empowerment.
• Opportunity to shape the future of wealth management using cutting-edge AI and MLOps tools.
• Collaboration within a professional Agile Release Train (ART) structure.