Senior Software Engineer (AI Evaluation)

TL;DR

Senior Software Engineer (AI Evaluation): Building high-quality datasets and virtual environments to evaluate the efficacy of AI coding agents with an accent on creating complex developer scenarios and rigorous evaluation criteria. Focus on designing isolated Linux environments, calibrating adversarial tasks, and analyzing the failure modes of frontier LLMs.

Location: Must be based in the European Union

Company

hirify.global AI creates data that powers leading GenAI models, utilizing a combination of experts, crowd, and a tech platform to teach AI models to reason.

What you will do

• Create isolated development environments emulating workstations with Linux, dev tools, and real web application codebases.
• Design complex prompts and tasks that mirror real-world developer tickets and conversations.
• Develop rigorous tests and evaluation criteria to verify AI agent solutions.
• Iterate with AI agents to refine tests and design adversarial scenarios to challenge frontier models.
• Review and analyze AI-generated code to identify weaknesses and success factors.

Requirements

• 8+ years of professional software engineering experience in any modern language.
• Proficiency and willingness to work primarily in Python for project tasks.
• Practical experience integrating coding agents (e.g., Claude Code, Codex) into engineering workflows.
• Strong understanding of Docker and modern development workflows.
• Ability to communicate complex technical reasoning and decisions clearly.

Culture & Benefits

• Freelance collaboration with a potential path into a permanent project team.
• Flexible, fully remote work schedule.
• Opportunity to work at the cutting edge of AI with industry-leading customers.
• Access to a dynamic community of global AI experts.