Senior Software Developer In Test (Python (AI))

TL;DR

Senior Software Developer in Test (Python (AI)): Building and maintaining automated testing frameworks and quality evaluation pipelines for AI/LLM systems with an accent on RAG, agentic workflows, and LLM observability. Focus on designing evaluation suites, performing adversarial testing, and optimizing AI system reliability.

Location: Remote (Worldwide)

Company

hirify.global is a global provider of software solutions and infrastructure for the iGaming industry.

What you will do

• Analyze requirements and define testing approaches for new features and AI/LLM-based systems.
• Develop and maintain automated tests using Python and PyTest.
• Build automated quality evaluation pipelines for AI systems using metrics and LLM-as-judge approaches.
• Evaluate agentic workflows, MCP servers, tool-call behavior, and non-deterministic system outputs.
• Perform adversarial testing, including prompt injection, jailbreaks, and data leakage risk assessment.
• Contribute to CI/CD improvements and maintain internal testing tools and mocks.

Requirements

• 5+ years of experience in Quality Assurance (manual and automation).
• Strong proficiency with Python-based test automation frameworks (PyTest, Behave).
• Experience with CI/CD and monitoring tools like Datadog, ELK, or Sentry.
• Familiarity with RAG, LLM evaluation metrics (groundedness, faithfulness), and non-deterministic systems.
• English: B2 level or higher.
• Location: Must be able to work remotely (Worldwide).

Nice to have

• Experience with load, stress, or performance testing.
• Knowledge of Kubernetes and Prompt Engineering.
• Experience with AI safety, guardrails, or curating evaluation datasets.
• Familiarity with LangChain, LangGraph, vector databases, or semantic search.

Culture & Benefits

• Private health insurance and a comprehensive Mental Health Program.
• Sports benefits and free online English lessons.
• Paid time off and maternity leave support.
• Upskilling opportunities through internal workshops and professional conferences.
• Referral program rewards.