Senior AI Systems Quality Engineer (AI)

TL;DR

Senior AI Systems Quality Engineer (AI): Building and evolving automated validation frameworks and testing platforms for agentic, LLM-driven healthcare data systems with an accent on reliability, non-deterministic behavior management, and production-grade quality gates. Focus on designing custom evaluation harnesses within Databricks and MLflow to ensure safe, scalable, and trustworthy AI deployments.

Location: Must be based in the US

Company

hirify.global is a health-tech company transforming healthcare data into a trusted foundation for better care and cost decisions through clean, connected, and reliable data platforms.

What you will do

• Build and ship production-grade automated validation frameworks and evaluation pipelines across the AI lifecycle.
• Design an AI testing platform integrated with Databricks and MLflow for repeatable testing and auditability.
• Create large-scale, scenario-based test suites to validate agentic workflows, edge cases, and failure modes.
• Define measurable quality signals for LLM systems and integrate them into CI/CD pipelines as automated quality gates.
• Partner with AI and platform teams to define system contracts, guardrails, and safe-degradation patterns.
• Own AI release readiness by establishing go/no-go criteria based on measurable quality thresholds.

Requirements

• 7+ years of software engineering experience, primarily in backend or platform systems.
• Proven experience designing and implementing AI testing automation in production environments.
• Strong proficiency in Python and/or TypeScript within modern AI engineering stacks.
• Hands-on experience with LLM-based or agentic workflows and non-deterministic behavior.
• Deep understanding of CI/CD integration and AWS cloud-native architectures.
• Must be based in the US.

Nice to have

• Experience with Databricks-native environments and Medallion architecture.
• Familiarity with LLM evaluation techniques, guardrails, and policy enforcement frameworks.
• Experience using observability tools like Datadog, Prometheus, or Grafana.
• Formal training or certification in AI/ML systems (e.g., ISTQB AI Testing, AWS ML Specialty).

Culture & Benefits

• Unlimited paid time off.
• Comprehensive health coverage with multiple plan options.
• Equity for every employee.
• Home office setup allowance and monthly cell phone allowance.
• Growth-focused environment with support for professional development.
• Flexible remote work policy.