Senior AI Systems Quality Engineer (AI)

Location: United States. Remote work is available, with a work-from-anywhere policy.

Company

hirify.global builds healthcare data platforms that help health plans create trusted, connected data foundations for better care and cost decisions.

What you will do

• Build production-grade automated validation frameworks, test harnesses, and evaluation pipelines across the AI lifecycle.
• Design and evolve an AI testing platform integrated with Databricks and MLflow for repeatable testing, lineage, traceability, and auditability.
• Create large-scale scenario-based test suites covering agentic workflows, edge cases, long-tail scenarios, and failure modes.
• Validate orchestration behavior, including tool use, memory, decision logic, and non-deterministic system behavior.
• Define guardrails, system contracts, safe-degradation patterns, and measurable quality signals for grounding, hallucinations, relevance, latency, and cost.
• Integrate automated quality gates into CI/CD pipelines and own release-readiness criteria for AI systems.

Requirements

• 7+ years of software engineering experience, primarily in backend or platform systems.
• Production experience designing AI testing automation and custom validation or evaluation frameworks for complex distributed systems.
• Strong proficiency in Python and/or TypeScript and hands-on experience with LLM-based or agentic workflows.
• Experience with large-scale regression testing, long-tail evaluation, non-deterministic outputs, drift detection, bias and fairness testing, and robust regression strategies.
• Deep understanding of CI/CD integration, AWS cloud-native architectures, and automated quality gates.
• Knowledge of security, privacy, operational risk, failure recovery, and PHI-safe behavior in regulated or mission-critical environments.

Nice to have

• Experience with Databricks-native environments and Medallion architecture.
• Experience using MLflow for model evaluation, lineage tracking, and auditability.
• Exposure to Datadog, Prometheus, or Grafana for distributed AI observability.
• Familiarity with LLM evaluation, guardrails, policy enforcement, prompt and agent behavior versioning, and AI-generated test scenarios.
• AI/ML testing or cloud certifications such as ISTQB AI Testing, AWS ML Specialty, or Google ML Engineer.

Culture & Benefits

• Remote-first flexibility with work-from-anywhere support.
• Unlimited paid time off.
• Comprehensive health coverage with multiple plan options.
• Equity grants for every employee and eligibility for performance bonuses.
• Home office setup allowance and monthly cell phone allowance.
• Inclusive, collaborative environment focused on healthcare impact and responsible use of AI.