Senior AI Engineer (LLM)

Location: Warsaw, Poland; flexible hybrid mode with 1–2 office days per week in Warsaw or Kraków.

Company

hirify.global is hiring for an AI-powered shopping assistant project.

What you will do

• Lead the development and technical ownership of an LLM evaluation platform supporting a production AI-powered shopping assistant.
• Develop LLM evaluation pipelines, LLM-as-a-judge solutions, evaluation datasets, golden test sets, and automated regression tests.
• Analyse real customer conversations, measure response quality, compare model performance, and identify hallucinations, incorrect responses, risks, and unusual user behaviour.
• Build monitoring and observability for AI systems using tools such as Langfuse, Braintrust, or Weave.
• Improve prompts, context management, AI agents, tool calling, agent workflows, guardrails, adversarial testing, and abuse detection.
• Collaborate with product managers and engineering teams to improve reliable and measurable LLM performance.

Requirements

• At least 5 years of software engineering experience and at least 2 years working with LLM systems in production.
• Strong Python development and hands-on experience evaluating LLM applications.
• Experience with LLM-as-a-judge systems, evaluation frameworks, prompt engineering, and context engineering.
• Experience with AI agents, tool calling, and multi-turn conversations.
• Experience with LLM observability tools such as Langfuse, Braintrust, Weave, or similar.
• SQL experience and experience working with large datasets.
• Experience with CI/CD, testing, Docker or other container technologies, and cloud infrastructure.
• Experience using AI development tools such as Cursor, Claude Code, or GitHub Copilot.
• Good English and Polish.

Nice to have

• Human evaluation and calibration of LLM results.
• Golden datasets and regression testing for generative AI.
• Agent orchestration and automated workflows.
• MCP and agent-to-agent integrations.
• AI red-teaming, jailbreak testing, and guardrails.
• Experience improving LLM applications using production feedback.
• Experience with e-commerce, search, or recommendation systems.
• Experience as a Technical Lead or Staff Engineer.

Culture & Benefits

• Flexible hybrid work with 1–2 office days per week in Warsaw or Kraków.
• Solid, competitive salary.
• Work in a multinational environment on international projects.
• Comprehensive healthcare.
• Long-term B2B contract with a stable project pipeline.