Senior QA Engineer (AI)

TL;DR

Senior QA Engineer (AI): Building the foundational quality assurance and evaluation framework for AI-powered banking software with an accent on LLM output validation, regression baselines, and CI/CD integration. Focus on designing non-deterministic testing strategies, ensuring regulatory compliance, and scaling quality processes for high-stakes financial environments.

Location: Must be based in the United States

Company

hirify.global builds purpose-built small language models and AI agents for financial institutions, operating under strict compliance and audit standards.

What you will do

• Design and execute evaluation frameworks for LLM and agentic AI outputs.
• Build and maintain automated test suites for backend APIs, document ingestion, and AI inference workflows.
• Implement and enforce quality gates within CI/CD pipelines.
• Define behavioral contracts and regression baselines for model performance.
• Produce test artifacts and audit logs to meet SOC 2 Type II compliance standards.
• Triage production bugs and own the regression testing process.

Requirements

• 7+ years of experience in software QA engineering.
• 2+ years of hands-on experience testing AI or ML systems.
• Fluent in Python with experience using pytest, Playwright, or Selenium.
• Proven experience with AI evaluation tools like RAGAS, DeepEval, or LangSmith.
• Must be based in the United States.
• Ability to trace failures across application layers, infrastructure, and REST APIs.

Nice to have

• Experience in fintech, banking, or other highly regulated industries.
• Familiarity with document processing pipelines and multi-agent architectures.
• Experience with observability tooling such as Arize or Langfuse.

Culture & Benefits

• Competitive base salary and meaningful equity package.
• Fully remote work environment within the US.
• Opportunities for occasional travel to client sites and team offsites.
• Impactful role building a QA function from the ground up in a high-growth startup.