Senior QA Analyst (AI)

TL;DR

Senior QA Analyst (AI): Leading testing efforts for AI-driven features and machine learning models across hirify.global’s platform with an accent on validating model accuracy, reliability, and data quality. Focus on designing and executing test strategies for LLM-based systems and behavior analysis pipelines, ensuring rigorous testing of nondeterministic systems.

Location: Remote from select locations or hybrid in El Dorado Hills, CA or Chicago, IL metro area.

Company

hirify.global Security is a fast-growing cybersecurity company transforming privileged access management with zero-trust and zero-knowledge solutions globally.

What you will do

• Design and execute test plans for AI and ML features, including classification, summarization, and anomaly detection.
• Develop Python-based automation and evaluation scripts to validate model performance and output quality.
• Test LLM-based features for accuracy, guardrail effectiveness, hallucination risk, and behavior under edge cases.
• Validate data pipelines, input/output transformations, and model integration points.
• Conduct regression testing on ML models and AI workflows to ensure reliability across releases.
• Ensure security, privacy, and zero-trust requirements for AI model interactions and data handling.

Requirements

• 3+ years of experience in QA, test automation, or evaluation of complex systems.
• Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs, or data pipelines.
• Understanding of machine learning concepts such as classification, summarization, or anomaly detection.
• Experience testing LLM-integrated features or AI-driven workflows.
• Familiarity with ML tooling such as Pytest, Hugging Face libraries, or Jupyter Notebooks.
• Experience with API testing, JSON validation, and test data generation.
• Ability to test nondeterministic systems and evaluate model quality with structured methodologies.
• Experience testing cloud-integrated applications or systems deployed on AWS.
• Bachelor’s degree in Computer Science, Information Systems, or a related field, or equivalent practical experience.

Nice to have

• Experience testing security-focused AI or threat analytics systems.
• Familiarity with ML observability tools, vector databases, or data processing frameworks.
• Knowledge of zero-trust principles and secure data handling.
• Experience working with LLM providers or AI guardrail frameworks.

Culture & Benefits

• Medical, Dental & Vision (inclusive of domestic partnerships).
• Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life.
• Voluntary Short/Long Term Disability Insurance.
• 401K (Roth/Traditional).
• A generous PTO plan that celebrates your commitment and seniority.
• Above market annual bonuses.