Principal AI Tool Engineer For Software Automation/Testing (Cybersecurity)

TL;DR

Principal AI Tool Engineer (AI/ML Testing): Designing and developing testing frameworks for AI/ML models and LLM applications with an accent on model validation, benchmarking, and hallucination detection. Focus on building automated pipelines for regression testing and creating evaluation datasets for edge cases.

Location: Must be based in Santa Clara, CA, USA. No immigration sponsorship available.

Salary: $147,000 - $237,500 per year

Company

hirify.global is a global leader in cybersecurity, dedicated to protecting the digital way of life through AI-driven innovation and disruptive technology.

What you will do

• Design and develop testing frameworks for AI/ML models and LLM applications.
• Build automated pipelines for model validation, regression testing, and benchmarking.
• Create evaluation datasets, synthetic data, and test scenarios for edge cases.
• Implement metrics to assess accuracy, robustness, latency, and safety.
• Develop tools for prompt testing, output validation, and hallucination detection.
• Collaborate with engineering and product teams to define test strategies and monitor production performance.

Requirements

• Bachelor’s or Master’s degree in Computer Science, AI, Machine Learning, or a related field.
• 8+ years of experience in software engineering, QA automation, or ML engineering.
• Strong programming skills in Python.
• Experience with testing frameworks (e.g., PyTest, unittest) and CI/CD pipelines.
• Familiarity with machine learning concepts and model evaluation techniques.
• No immigration sponsorship available for this position.

Nice to have

• Experience with LLMs, prompt engineering, and frameworks like LangChain or OpenAI Evals.
• Knowledge of AI safety, bias detection, and adversarial testing.
• Experience with cloud platforms such as AWS, GCP, and Azure.
• Exposure to synthetic data generation and simulation environments.

Culture & Benefits

• Mission-driven environment focused on cybersecurity innovation.
• Strong emphasis on in-person collaboration at the headquarters.
• Competitive compensation package including restricted stock units and bonuses.
• Commitment to diversity, inclusion, and equal opportunity employment.