Software Engineer, Agent Evaluation and Harness (AI)

Location: Stockholm, Sweden

Company

hirify.global is an AI-native data and intelligence platform that unifies marketing and business data for agencies and in-house teams.

What you will do

• Build the agent harness, including orchestration, tool use, structured outputs, fallback routing, guardrails, and cost and latency tracking.
• Tune prompts, retrieval, and grounding for chat, Studio, and autonomous background agents working across customer data.
• Design evaluation systems with curated datasets, offline replay, judges, scorers, regression alerts, and dashboards.
• Build feedback loops from real user signals and create debugging tools for tracing agent failures and identifying recurring patterns.
• Improve retrieval through chunking, embeddings, context-window management, and grounding answers in live SQL results.
• Develop MCP connectors and assess model releases, provider changes, and fine-tuning opportunities.

Requirements

• Production experience shipping LLM applications used by real users.
• Experience building or operating evaluation systems, datasets, scorers, regression suites, experimentation platforms, or comparable quality tooling.
• Strong familiarity with LLM APIs including OpenAI, Anthropic, and Gemini, as well as prompting, function calling, structured outputs, and modern agent patterns.
• Ability to manage context, design for prompt injection and hallucination risks, and address latency and cost constraints.
• Strong software engineering fundamentals with fluency in Python and TypeScript, or willingness to learn one quickly.
• Full-time role based in Stockholm, Sweden.

Culture & Benefits

• Freedom to choose tools and approaches with responsibility for shipping substantial work.
• Large token budget and access to advanced development tools.
• Competitive salary and meaningful equity.
• Direct collaboration with the co-founders.