Senior Product Software Engineer - AI Quality Engineering

Location: Cape Girardeau, Missouri, United States. Applicants may be required to appear onsite at a hirify.global office during the hiring process.

Salary: $83,400–$145,650 USD per year, plus bonus eligibility.

Company

hirify.global develops professional tax and accounting products and AI-powered experiences.

What you will do

• Own the quality engineering strategy for enterprise generative AI applications, including RAG pipelines, LLM orchestration, agentic workflows, APIs, ingestion pipelines, and AI-driven user experiences.
• Design evaluation harnesses and automated regression suites measuring retrieval accuracy, citation correctness, relevance, groundedness, hallucination rate, safety, latency, cost, and end-to-end behavior.
• Build Python-based test automation for APIs, services, chat workflows, data pipelines, and AI evaluation workflows within CI/CD pipelines.
• Partner with engineering, product, UX, and subject-matter teams to validate integrations, observability, resilience, security, and production readiness.
• Lead root-cause analysis, define quality gates, and advise on release readiness and tradeoffs involving accuracy, reliability, latency, cost, and user trust.
• Establish observability dashboards and mentor engineers in AI evaluation, non-deterministic testing, automation, and production-quality practices.

Requirements

• 5+ years of experience in quality engineering, SDET, software engineering, test automation, reliability engineering, or equivalent roles.
• 2+ years of hands-on experience testing or evaluating AI/ML, LLM, RAG, or other non-deterministic systems in production or production-like environments.
• Strong Python test automation skills with Pytest, unittest, or equivalent frameworks, including reusable test harnesses and regression suites.
• Experience validating REST APIs, microservices, service integrations, data ingestion pipelines, and end-to-end web or chat workflows.
• Practical knowledge of RAG systems, vector search, embeddings, prompt behavior, model evaluation, groundedness, citation quality, and AI failure modes.
• Knowledge of CI/CD, Git-based workflows, observability, logging, tracing, dashboards, metrics, responsible AI, security, privacy, fairness, and customer impact.

Nice to have

• Experience with LLM observability and evaluation platforms such as LangSmith, Langfuse, TruLens, Arize/Phoenix, or custom tooling.
• Experience with vector databases, search platforms, LLM application frameworks, Azure AI services, Docker, Kubernetes, and cloud-native delivery.
• Experience with performance, load, resiliency, and production observability tools such as Locust, k6, JMeter, Grafana, Prometheus, Datadog, or OpenTelemetry.
• Knowledge of LLMOps, model monitoring, A/B testing, human-in-the-loop evaluation, and production feedback loops.
• Experience with sensitive or regulated data, or backgrounds in tax, accounting, legal, financial, or other accuracy-critical domains.

Culture & Benefits

• Work in an Agile environment with product, design, engineering, and domain expert teams.
• Medical, dental, and vision plans.
• 401(k), FSA/HSA, commuter benefits, and tuition assistance.
• Vacation, sick time, and paid parental leave.
• Interviews must be completed without AI tools or external prompts; the process includes in-person interviews and may require onsite attendance.