AI Quality Engineer (AI)

TL;DR

AI Quality Engineer (AI): Designing and implementing evaluation frameworks to assess LLM and agentic AI system quality with an accent on accuracy, safety, and task completion rates. Focus on building automated test pipelines for agentic workflows, detecting model regressions, and defining quality metrics for generative AI.

Location: Atlanta, GA, US. Must be eligible to work in the United States without sponsorship.

Company

Cloud-based software provider supporting purpose-driven nonprofits and associations globally to simplify operations and grow revenue.

What you will do

• Design and implement evaluation frameworks (evals) for LLM and agentic AI systems.
• Build automated test pipelines covering unit, integration, and end-to-end scenarios across agentic workflows.
• Develop tooling to detect regressions in model behavior and prompt outputs across releases.
• Define and track AI quality metrics including hallucination rates, tool-use accuracy, and latency.
• Collaborate with product and engineering teams to identify edge cases and failure modes.
• Contribute to prompt evaluation strategies, including red-teaming and bias assessments.

Requirements

• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.
• 3–5 years of professional software or quality engineering experience.
• Hands-on experience with LLMs (GPT-4, Claude, Gemini) or agentic AI systems.
• Proficiency in Python for scripting, test automation, and data analysis.
• Experience designing and running evaluations (evals) for generative AI features.
• Must be eligible to work in the United States without sponsorship.

Nice to have

• Experience with prompt engineering and systematic evaluation methodologies.
• Familiarity with AI safety, alignment, and guardrails.
• Exposure to agentic orchestration frameworks like LangChain, LangGraph, AutoGen, or CrewAI.
• Experience with vector databases or RAG pipelines.
• Knowledge of AI observability tools such as LangSmith, Weights & Biases, or Arize.

Culture & Benefits

• Medical, Dental & Vision benefits.
• 401(k) Savings Plan with company match.
• Flexible planned paid time off and generous sick leave.
• Remote work flexibility and commitment to work-life balance.
• Employer-paid parental leave and short-term disability.