Product Data Scientist (AI Evaluation & Quality)

TL;DR

Product Data Scientist (AI Evaluation & Quality): Building and maintaining the evaluation loop for AI products including financial co-pilots and voice agents with an accent on pre-launch quality gates and post-launch monitoring. Focus on creating offline eval suites, building online quality dashboards, and translating ML metrics into product decisions.

Location: Remote within the EU/Europe

Company

European fintech startup providing an all-in-one financial B2B solution for entrepreneurs, integrating banking, accounting, and invoicing.

What you will do

• Own and extend the offline eval suite across products, including datasets, judges, and metrics.
• Build and maintain online quality dashboards monitoring resolution rate, CSAT, and LLM-as-judge signals.
• Close the production feedback loop by mining failure patterns from real traffic and proposing fixes.
• Harden evaluation methodology regarding judge stability and non-determinism handling.
• Translate data insights into business decisions and clear trade-offs during weekly syncs.

Requirements

• 3+ years of experience in analyst or data scientist roles, with at least one year in a product context.
• Proficiency in Python and SQL for end-to-end analysis.
• Solid foundation in statistics, including sampling, hypothesis testing, and variance.
• Analytical mindset capable of starting from a business question rather than a tool.
• Must be based in or able to work within the EU/Europe.

Nice to have

• Experience in quality analytics for ML systems such as ranking, recommendations, or classification.
• Hands-on experience evaluating LLM applications (RAG, agents, tool use).
• Experience building LLM agents through side projects or personal experiments.

Culture & Benefits

• AI-first development environment with Claude Code as the default authoring tool.
• Stock Options Program available to every team member.
• Flexibility to work remotely or in a hybrid model across Europe.
• Exclusive "Work & Swim" program: one month in a corporate apartment in Cyprus.
• Modern, friendly, and eco-conscious corporate culture.