Lead Quality Engineer (AI)

TL;DR

Lead AI Quality Engineer (AI): Designing and implementing evaluation harnesses for AI-powered product experiences with an accent on measuring retrieval accuracy, citation correctness, and overall response quality. Focus on developing automated tests for APIs and ingestion pipelines, identifying root causes of AI-driven response failures, and ensuring testability is built into features from day one.

Location: Hybrid role, requiring 2 days a week in a hirify.global office in the USA (Coppell, TX; Riverwoods, IL; Kennesaw, GA; Chicago, IL; Houston, TX)

Salary: $89,600.00 – $157,000.00 USD

Company

hirify.global is a product company focused on delivering innovative professional tax and accounting software solutions.

What you will do

• Design and implement evaluation harnesses to measure AI system behavior and quality.
• Develop automated tests for APIs, ingestion pipelines, and chat workflows.
• Collaborate with developers and product managers to define AI quality metrics (accuracy, latency, hallucination rate).
• Analyze logs, traces, and feedback signals to identify root causes of failures in AI-driven responses.
• Create regression suites to ensure changes to prompts, chunking, or embeddings don’t break existing behavior.
• Validate REST APIs and service integrations for resilience, correctness, and security.

Requirements

• Bachelors Degree in Computer Science or equivalent.
• 5+ years of experience in software testing, quality engineering, or equivalent engineering roles with a focus on validation and reliability.
• Experience with AI evaluation frameworks (e.g., LlamaIndex evals, OpenAI Evals, Ragas, TruLens, or custom harnesses).
• Strong skills in Python testing frameworks (Pytest, unittest, or equivalent).
• Experience testing web applications and APIs.
• Familiarity with AI/ML or non-deterministic system testing.
• Knowledge of CI/CD pipelines, Git, and automated regression testing.
• Strong analytical skills: able to define metrics and success criteria where outputs aren’t deterministic.
• Comfortable working in a fast-paced Agile environment with weekly sprints, pairing, and close collaboration with PM/UX/Dev.

Nice to have

• Knowledge of retrieval-augmented generation (RAG) pipelines.
• Experience with metrics/observability tooling (Grafana, Prometheus, Datadog).
• Familiarity with containerized environments (Docker, Kubernetes).
• Exposure to performance/load testing tools (Locust, k6, JMeter).

Culture & Benefits

• Guided by core values: Focus on Customer Success, Make it Better, Aim High and Deliver, and Win as a Team.
• Committed to driving success for customers by delivering innovative solutions that exceed expectations.
• Continuously strive to improve processes and products, aiming for excellence.
• Collaboration and teamwork are central to the culture, enabling great results together.
• Offers a wide variety of competitive benefits including Medical, Dental, & Vision Plans, 401(k), FSA/HSA.
• Commuter Benefits, Tuition Assistance Plan, Vacation and Sick Time, and Paid Parental Leave.

Hiring process

• Candidates are asked to participate in interviews without the assistance of AI tools or external prompts.
• To maintain integrity, candidates should remove virtual backgrounds, and in-person interviews may be included.
• Use of AI-generated responses or third-party support during interviews will be grounds for disqualification.
• Applicants may be required to appear onsite at a hirify.global office as part of the recruitment process.