Test Lead (AI)

Location: Hybrid at hirify.global HQ in Athlone, Ireland

Company

hirify.global partners with telecommunications and financial services organizations to modernize legacy systems, migrate mission-critical platforms to the cloud, and build AI-driven automation across the EU, UK, and US.

What you will do

• Define and own QA strategy, quality gates, acceptance criteria, offline evaluation, regression testing, and production monitoring for agentic applications.
• Validate Network Digital Twin fidelity against real networks and use it as a sandbox for safely testing agents.
• Build evaluation harnesses, golden datasets, automated tests, and scenario-based tests for agents, prompts, RAG pipelines, tools, APIs, and multi-agent workflows.
• Test tool and API calls, including MCP and function calling, for correctness, sequencing, error handling, permissions, safety, and guardrail bypasses.
• Automate REST and event-driven microservice tests in CI/CD, and debug failures using traces, logs, metrics, and observability tools.
• Lead and mentor QA engineers, coach technical teams on AI testing, report quality and regression metrics, and improve testing practices through agile collaboration.

Requirements

• 7+ years of software testing experience, including at least 2 years leading QA for a team or product area.
• Strong experience testing distributed microservices, REST APIs, and event-driven systems such as Kafka or Pub/Sub.
• Hands-on experience testing LLM-based or agentic applications and evaluating non-deterministic outputs.
• Practical knowledge of golden datasets, LLM-as-judge, trajectory and tool-call evaluation, and regression testing across model or prompt changes.
• Experience with at least one evaluation or observability framework such as DeepEval, Ragas, promptfoo, LangSmith, or Langfuse.
• Solid Python or similar coding skills for test automation and evaluation pipelines, including tools such as Pytest.
• Understanding of AI risks and testing for hallucination, prompt injection, bias, data leakage, unsafe actions, and guardrail bypass.
• Experience embedding automated tests and quality gates into CI/CD pipelines.
• Ability to debug backend and agent interactions using logs, traces, and metrics, and explain AI quality risks to technical and non-technical stakeholders.

Nice to have

• Experience testing Digital Twins, simulators, or network emulation environments.
• Familiarity with telecommunications, Passive Optical Networks, and network operations use cases.
• Experience with Google Cloud Platform, including Vertex AI, GKE, Pub/Sub, or Cloud Run.
• Knowledge of LangGraph, Google ADK, CrewAI, or the Model Context Protocol.
• Performance and load testing of AI services, including token cost and latency budgets.

Culture & Benefits

• Full-time, permanent employment.
• Hybrid work from the hirify.global HQ in Athlone, Ireland.
• Work in an inclusive environment focused on diversity, belonging, innovation, and collaboration.
• Support for local community initiatives promoting equity and opportunity.