Senior Data Scientist (AI Evaluation & Quality)

Location: Vilnius; remote or hybrid work across Europe.

Company

hirify.global is a European fintech startup building an all-in-one B2B financial platform for entrepreneurs, combining banking, accounting, financial management, and invoicing.

What you will do

• Own and extend offline evaluation suites across AI products, including datasets, judges, metrics, capability tests, and regression tests.
• Build and maintain online quality dashboards covering resolution rate, CSAT, user feedback, LLM-as-judge signals, error rate, and latency.
• Analyze real-traffic failures, identify recurring patterns, and convert findings into regression cases.
• Collaborate with AI engineers, Product, and domain experts to propose product and quality improvements.
• Improve evaluation methodology by addressing judge stability and non-deterministic results.
• Translate analytical findings into clear trade-offs and product decisions during weekly syncs.

Requirements

• 3+ years of experience in analyst or data scientist roles, including at least one year in a product context.
• Strong Python and SQL skills for end-to-end analysis.
• Solid statistical foundation, including sampling, hypothesis testing, variance, and noisy metrics.
• Analytical approach focused on business questions and measurable decisions.
• Fluency with AI-assisted coding or genuine readiness to become fluent quickly.
• Ability to work remotely or in a hybrid model across Europe.

Nice to have

• Quality analytics experience for ML systems such as ranking, recommendations, or classification.
• Hands-on experience evaluating LLM applications, including RAG, agents, tool use, or judges.
• Experience building LLM agents through side projects or personal experiments.

Culture & Benefits

• AI-assisted coding is the default authoring environment, with Claude Code used for SQL, Python, analyses, dashboards, and internal scripts.
• Opportunities for continuous personal and professional development.
• Stock options available to every team member.
• Supportive, friendly, and eco-conscious corporate culture.
• Work & Swim program offering one month in a corporate apartment in Cyprus.