Senior AI Quality & Evaluation Engineer

Location: Hybrid in Sofia or Ruse, Bulgaria

Company

International engineering firm helping businesses operate with AI at the core across industries, technologies, and markets.

What you will do

• Design and own automated test strategies covering unit, integration, contract, end-to-end, and load testing, integrated with CI/CD.
• Build and maintain evaluation suites with datasets, rule-based and LLM-as-judge scorers, and regression tests for prompt, model, and tool changes.
• Evaluate models and tools against quality, safety, latency, and cost criteria before deployment.
• Perform adversarial testing for prompt injection, jailbreaks, data leakage, and tool misuse; maintain red-team suites and report findings with severity and reproduction steps.
• Define quality gates and release criteria, expose quality in dashboards, monitor production behavior, and raise alerts when quality degrades.
• Provide quality evidence for security reviews, AI governance requirements, and clients; coach engineers on testable design and evaluation coverage.

Requirements

• 5+ years of experience in software quality or test engineering.
• Strong Python skills and experience writing test frameworks, not only test cases; C#/.NET is a plus.
• Hands-on experience evaluating LLM applications using frameworks such as Promptfoo, DeepEval, Ragas, Langfuse, or LangSmith, including dataset creation, judge design, and metric design.
• Working knowledge of RAG and agent architectures, including retrieval quality, tool-call errors, loops, hallucinations, and context loss.
• Experience with adversarial or security testing of AI systems or software systems generally.
• CI/CD and infrastructure-as-code literacy, with experience running tests on at least one of Azure, AWS, or GCP.
• Clear written reporting skills for turning findings into decisions.
• Full working proficiency in English.

Nice to have

• Familiarity with the EU AI Act, ISO/IEC 42001, or the NIST AI RMF.
• Experience with performance and load testing of LLM-backed services.
• Experience benchmarking open-weight models.
• Experience in regulated industries or client-facing delivery.

Culture & Benefits

• Flexible hybrid work arrangements with laptops and equipment provided.
• Private health insurance and well-being programs.
• Training opportunities and support for professional growth.
• Social events and access to office facilities.
• Respectful, inclusive environment with equal opportunities.

Hiring process

• Submit a CV and discuss skills and career goals during a talent call.
• Complete a technical interview focused on experience, strengths, and approach to work.
• Some roles include a customer interview before an offer.