Senior Product Software Engineer - AI Quality Engineering

Location: USA - Cape Girardeau, Missouri. Applicants may be required to appear onsite at a hirify.global office during the recruitment process.

Salary: $83,400.00–$145,650.00 USD annually, plus bonus eligibility.

Company

hirify.global provides professional information, software, and services for tax, accounting, and other specialized domains.

What you will do

• Own the quality engineering strategy for enterprise generative AI applications, including RAG pipelines, LLM orchestration, agentic workflows, APIs, ingestion pipelines, and AI-driven user experiences.
• Architect evaluation harnesses and automated regression suites measuring retrieval accuracy, citation correctness, relevance, groundedness, hallucination rate, safety, latency, cost, and end-to-end behavior.
• Develop Python-based test automation for APIs, services, chat workflows, data pipelines, and AI evaluation workflows within CI/CD pipelines.
• Define quality gates and advise product and engineering leaders on release readiness, risk, reliability, cost, latency, and user trust.
• Investigate AI quality failures using logs, traces, evaluation outputs, feedback signals, customer scenarios, and domain-specific acceptance criteria.
• Build observability dashboards and mentor engineers in AI evaluation, non-deterministic testing, automation design, and production-quality practices.

Requirements

• 5+ years of experience in quality engineering, SDET, software engineering, test automation, reliability engineering, or equivalent roles.
• 2+ years of hands-on experience testing, evaluating, or validating AI/ML, LLM, RAG, or other non-deterministic systems in production or production-like environments.
• Strong Python test automation skills with Pytest, unittest, or equivalent frameworks, including reusable test harnesses and regression suites.
• Experience validating REST APIs, microservices, service integrations, data ingestion pipelines, and end-to-end web or chat workflows.
• Practical knowledge of RAG systems, vector search, embeddings, prompt behavior, model evaluation, groundedness, citation quality, and AI failure modes.
• Knowledge of CI/CD, Git-based workflows, release gates, observability, logging, tracing, dashboards, metrics, security, privacy, fairness, and responsible AI practices.

Nice to have

• Experience with LangSmith, Langfuse, TruLens, Arize/Phoenix, or custom LLM observability and evaluation tooling.
• Experience with Azure AI Search, Pinecone, Weaviate, Elasticsearch, Azure OpenAI Service, Azure AI Studio, or other Azure AI services.
• Experience with LangChain, LlamaIndex, Semantic Kernel, Docker, Kubernetes, Locust, k6, JMeter, Grafana, Prometheus, Datadog, or OpenTelemetry.
• Knowledge of LLMOps, model monitoring, A/B testing, human-in-the-loop evaluation, or production feedback-loop design.
• Experience with sensitive, regulated, or professional-domain data, particularly in tax, accounting, legal, or financial contexts.

Culture & Benefits

• Collaboration with software engineers, product managers, UX professionals, subject matter experts, and other stakeholders.
• Medical, dental, and vision plans.
• 401(k), FSA/HSA, commuter benefits, and tuition assistance.
• Vacation, sick time, and paid parental leave.
• Fair and structured interviews focused on individual skills, experience, and communication.

Hiring process

• Interviews must be completed without AI tools, external prompts, or third-party assistance.
• Virtual backgrounds must be removed, and in-person interviews may be included.
• Applicants may be required to appear onsite at a hirify.global office.