AI Test Architect (AI)

TL;DR

AI Test Architect (AI): Architect a comprehensive "Quality Intelligence" platform using generative AI for automated test creation, self-healing execution, predictive defect analytics, and validation of AI features with an accent on LLM evaluation frameworks, ethical testing, and CI/CD integration. Focus on designing scalable benchmarks, red teaming protocols, observability pipelines, and self-healing automation to ensure reliability of agentic AI at scale.

Location: Fully remote position located in Colombia (Bogotá or Medellín).

Company

hirify.global is a leading cloud-native SaaS Fintech company providing audit and accounting software to over 500,000 users in 130 countries.

What you will do

• Architect AI-driven "Quality Intelligence" platform for test generation, self-healing, defect prediction, and SDLC integration.
• Design LLM/agent evaluation frameworks with benchmarks, red teaming, adversarial testing, and tools like LangSmith, DeepEval, RAGAS.
• Build infrastructure for AI testing in GitHub CI/CD, self-healing with Playwright/Cypress, synthetic data generation.
• Lead cross-functional collaboration, mentor QA teams, evangelize AI quality practices and metrics dashboards.
• Implement observability for AI telemetry, drift detection, feedback loops for model iteration.

Requirements

• Located in Colombia for fully remote work.
• 8+ years in Quality Engineering/Test Architecture in cloud-native SaaS, 2+ years in AI/ML/LLM testing.
• Deep expertise in AWS (serverless, IaC with Terraform), GitHub CI/CD.
• Proficiency in LLM apps (LangChain/LangGraph/LangSmith), automation (Playwright/Cypress), self-healing AI plugins.
• Strong programming in JavaScript/TypeScript/Python; AI concepts (transformers, RAG); LLM eval tools (DeepEval, RAGAS, Langfuse).
• Strong English communication skills; leadership in mentoring and standards definition.
• Bachelor's/Master's in CS/AI/ML or equivalent.

Nice to have

• Red teaming tools (Cobalt Strike, Nmap), adversarial testing methodologies.
• Relevant certifications.

Culture & Benefits

• Contrato a término indefinido with legal benefits, prepaid medicine, life insurance, internet/home office stipends.
• Competitive above-market compensation, 100% remote, excellent work-life balance.
• Training budget, mentorship, 5 PTO days, sick leave top-up, recognition awards, upgraded vacation after 5 years.
• Collaborative inclusive culture promoting innovation, independence, trust, global opportunities.