Model Evaluation Researcher

Location: Freiburg, Germany; hybrid attendance required at least 2 days per week or one full week every other week

Base compensation: €150K–€220K plus equity, depending on experience

Company

Black Forest Labs develops foundational generative AI technologies including Latent Diffusion, Stable Diffusion, and FLUX models for image, video, audio, and action generation.

What you will do

• Build and maintain the evaluation stack used for model checkpoints, finetuning runs, reinforcement learning runs, and competitor comparisons.
• Develop automated benchmarks, LLM-as-judge pipelines, and human evaluation studies through external panels.
• Design evaluations for prompt adherence, editing fidelity, aesthetics, text rendering, subject and character consistency, and safety.
• Measure agreement between human and automated evaluations, monitor stability over time, and improve unreliable evaluation methods.
• Benchmark FLUX against frontier models and identify quality gaps that aggregate metrics fail to capture.
• Define target metrics with research, reinforcement learning, and finetuning teams while developing new signals through APIs and MCP.

Requirements

• Advanced degree in a relevant field or equivalent practical experience.
• Hands-on experience evaluating generative models, including diffusion or flow-matching models or comparable modalities.
• Very strong Python and software engineering skills, with experience building reliable, maintainable pipelines.
• Working knowledge of PyTorch and modern machine learning infrastructure and practices.
• Strong understanding of statistics, sample sizes, inter-annotator agreement, confidence levels, and the limitations of evaluation metrics.
• Practical experience with LLM-as-judge systems, modern benchmarking, standard datasets, and critical analysis of model outputs.

Nice to have

• Experience designing and running human evaluations through panels and analyzing annotator disagreement and performance.
• Experience connecting evaluation with reward models, preference data, RLHF, DPO, or similar training methods.
• Experience evaluating agentic or multi-step systems.
• Experience building useful internal tools or making public contributions through papers or open source.

Culture & Benefits

• Research environment focused on scientific rigor, deep understanding, and high-quality products.
• Low-ego culture where ideas matter more than status and credit is shared.
• In-person collaboration supported through real offices and covered reasonable travel costs.
• Values centered on ambition, practical execution, empathy, and kindness.
• Equity included in the compensation package depending on experience.

Hiring process

• The details of the required in-person schedule will be discussed during the interview process.