Senior AI Software Developer in Test (AI)

TL;DR

Senior AI Software Developer in Test (AI): Pioneering AI-driven testing practices for cloud-native SaaS platform with intelligent agentic features, evolving quality strategy for LLM-based and agentic systems. Focus on designing deterministic/statistical testing for hallucinations, drift, bias, safety risks, building automated evaluation pipelines, and integrating AI quality signals into CI/CD for faster releases and compliance.

Fully remote position located in Colombia (Bogotá or Medellín).

hirify.global

Canada-based Fintech leader in audit and accounting software with 500,000+ users across 130 countries, at forefront of AI adoption in cloud-native SaaS platform.

What you will do

• Evolve AI-first quality strategy for SaaS architecture, integrate AI enhancements into CI/CD pipelines for predictive flakiness detection, automated test generation, self-healing scripts.
• Design testing approaches for non-deterministic LLM/agentic systems, build evaluation pipelines for correctness, faithfulness, retrieval, generation accuracy, tool-calling, multi-agent flows.
• Develop test frameworks for full AI lifecycle (prompts, datasets, embeddings, RAG, guardrails), implement red-teaming, bias checks, compliance mechanisms.
• Partner with product, data science, AI engineering teams for end-to-end testing, multi-agent simulations, knowledge sharing on AI testing practices.
• Drive metrics (DORA, test coverage, hallucination rate, drift detection), build dashboards, support A/B testing, post-deployment monitoring.
• Mentor SDETs, lead workshops, champion ethical AI mindset, define roadmaps for AI quality assurance.

Requirements

• 7+ years in Quality Engineering/SDET in cloud-native SaaS, 2+ years hands-on with AI/ML/LLM systems
• Expertise in automated testing infrastructure, CI/CD (Jenkins/GitHub Actions), test pyramid (unit to E2E)
• Strong full-stack testing (frontend/backend/API), proven experience testing LLMs, AI agents, RAG pipelines, risks (hallucinations, prompt injection, bias, drift)
• Proficiency in JS/TS, working knowledge of Python/Java; AI evaluation frameworks (Ragas, DeepEval, LangChain/LangSmith/LangFuse)
• Knowledge of observability (NewRelic), statistical testing, red-teaming, ethical AI practices
• Strong English language communication and collaboration skills
• Bachelor's/Master's in Computer Science/AI or related

Nice to have

• Performance/stress/load testing tools (K6, JMeter, Blazemeter)
• Certifications (e.g., ISTQB AI Testing)
• Strong portfolio (open-source, case studies)

Culture & Benefits

• Contrato a término indefinido with legal benefits: prepaid medicine, life insurance, funeral assistance
• Internet allowance, home office stipend, competitive compensation above market average
• 100% remote work, excellent work-life balance, 5 personal time off days, sick leave top-up, upgrade vacation after 5 years
• Training budget, mentorship, recognition awards, collaborative inclusive culture promoting innovation, trust, accountability