AI Quality Analyst (LLM)

TL;DR

AI Quality Analyst (LLM): Designing and implementing automated evaluation frameworks and pipelines for LLMs and agent graphs with an accent on RAG precision, tool-calling accuracy, and hallucination mitigation. Focus on integrating automated testing gates into CI/CD pipelines and optimizing model performance for latency and cost.

Location: Remote within Spain, Estonia, Greece, Poland, Portugal, United Kingdom, or Cyprus

Company

hirify.global is a fintech company providing business banking and financial management solutions for SMEs.

What you will do

• Architect and maintain scalable evaluation pipelines (Evals) using tooling such as LangSmith, DeepEval, Ragas, or Opik.
• Curate gold-standard datasets and synthetic evaluation profiles to reflect real-world business scenarios.
• Define and monitor quality KPIs for multi-agent workflows, specifically targeting intent-recognition safety and RAG precision.
• Lead failure and hallucination analyses using LLM-as-Judge patterns and guardrail heuristics.
• Partner with MLOps and Backend teams to integrate automated testing gates into CI/CD deployment pipelines.
• Analyze and optimize prompt templates and model selections to balance execution throughput, latency, and compute costs.

Requirements

• Strong software engineering fundamentals and concrete experience coding in Python.
• Deep intuition for non-deterministic LLM testing, including token behaviors and retrieval dynamics.
• Expertise in prompt engineering and identifying LLM failure states.
• Ability to work autonomously and coordinate with AI leads and product stakeholders.
• Must be based in Spain, Estonia, Greece, Poland, Portugal, UK, or Cyprus.

Culture & Benefits

• Opportunity to make a genuine impact on a scaling product.
• Ability to work within the European Union.
• Stock options eligibility.
• Unique "Work & Swim" program.
• Unwavering support and a care-oriented work environment.