Senior AI Quality Engineer (AI)

TL;DR

Senior AI Quality Engineer (AI/AWS): Designing and integrating testing guardrails for AI-powered agentic applications with an accent on reliability, observability, and safety in mission-critical environments. Focus on building bespoke evaluation harnesses and solving failure modes in non-deterministic AI workflows.

Location: Remote (Must be a U.S. Citizen and eligible for security clearance)

Salary: $122,000 - $177,400 per year

Company

A mission-driven technology company focused on strengthening national security and supporting critical industries through advanced sensors, autonomous systems, and AI-enabled software.

What you will do

• Design and integrate testing guardrails directly into agentic workflows and validate orchestration logic.
• Build custom evaluation harnesses and simulators tailored to how AI systems actually behave.
• Develop scenario-driven validation systems to test realistic edge cases and failure conditions.
• Architect agentic systems so correctness and safety are enforced by design.
• Implement cost-conscious evaluation strategies including hallucination detection and model relevancy.
• Collaborate with AI engineers and stakeholders to translate mission needs into concrete success criteria.

Requirements

• 5+ years of professional software engineering experience.
• Demonstrated experience building bespoke validation or testing frameworks using tools like Playwright, pytest, k6, or Guardrails AI.
• Hands-on experience designing and operating AI-powered systems, specifically agentic workflows or orchestrated LLM applications.
• Solid understanding of Amazon Web Services (AWS) and its native AI/ML services.
• Must be a U.S. Citizen and able to obtain and maintain a U.S. Security Clearance.
• Practical experience integrating automation into CI/CD pipelines in an agile environment.

Nice to have

• Proficiency in TypeScript, JavaScript (React, Next.js), and Python.
• Familiarity with prompt engineering and LLM evaluation techniques for non-deterministic behavior.
• Experience with observability tooling such as Datadog, Prometheus, or Grafana.
• Background in DoD environments, including experience with FedRAMP, CMMC, or Air Force programs.
• Ability and willingness to travel within the US as needed.

Culture & Benefits

• Comprehensive Medical, Dental, and Vision insurance.
• 401k matching.
• Paid Time Off and Company Holidays.
• Optional HSA and FSA.
• Base and Voluntary Life Insurance, plus Short-Term and Long-Term Disability.
• Employee Assistance Program.