AI Quality & Evaluation Engineer

Location: Naples, Italy; hybrid working model with 2 days per week remotely. Right to work in Italy required.

Gross annual salary: €40,000–€47,000 gross per year, paid over 14 monthly installments.

Company

hirify.global is a luxury cruise brand within MSC Cruises, operating in the travel and hospitality sector.

What you will do

• Build and maintain evaluation harnesses, datasets, LLM-as-judge evaluators, retrieval checks, and tool-call assertions for MSC AI services.
• Implement and calibrate approved judging criteria, analyse variance, and maintain scheduled production-canary evaluations with synthetic user personas.
• Develop adversarial and red-team test suites covering policy breaches, prompt injection, data disclosure, persona breaks, jailbreaks, and off-topic behaviour.
• Run automated checks supporting prompt, model, and retrieval release decisions, and integrate quality controls into continuous-integration pipelines.
• Build dashboards, trend analysis, and alerts; convert production incidents and reported failures into regression cases.
• Document evaluation methods, criteria, and results, and provide evidence for engineering, release, AI-governance, and regulatory decisions.

Requirements

• English is the primary working language.
• Degree in Computer Science, Information Technology, Engineering, or a related field.
• 4–7 years of total professional experience, including software engineering, test automation, or quality engineering experience with Python.
• Strong Python development skills; this is a hands-on engineering role focused on building and maintaining evaluation tooling.
• Practical experience testing or evaluating LLM-based systems, including LLM-as-judge approaches and non-deterministic output.
• Understanding of production AI failures, including hallucinations, retrieval gaps, prompt injection, jailbreaks, and unsafe or off-brand output.
• Experience designing and maintaining test datasets and regression suites as products change.
• Experience integrating automated testing into continuous-integration pipelines.
• Analytical, communication, and interpersonal skills, including presenting quality evidence and constructively challenging engineering and product decisions.

Nice to have

• Familiarity with LangChain, LangGraph, or the Microsoft Agent Framework.
• Experience with Langfuse, DeepEval, or comparable LLM tracing and evaluation tools.
• Experience evaluating retrieval-augmented generation systems, including retrieval-quality measurement.
• Experience with adversarial testing or AI red-teaming.
• Familiarity with AI-governance or regulatory regimes such as the EU AI Act.
• Background in travel, hospitality, or another large-scale consumer business.

Culture & Benefits

• Permanent employment under the applicable Italian collective labour agreement for shore-based personnel in private shipping companies.
• Hybrid work with 2 remote days per week and flexible working hours.
• €8 meal vouchers for each on-site working day.
• Welfare initiatives, productivity award, health insurance, learning opportunities, Employee Assistance Programme, and MSC group discounts and partnerships.
• Supportive environment focused on wellbeing, continuous learning, inclusive career growth, and diverse perspectives.