AI Automation Engineer

Location: Vienna, VA, United States; hybrid work model with onsite and remote work.

Company

hirify.global delivers outsourced services and workforce solutions across North America.

What you will do

• Design, develop, and maintain testing and evaluation frameworks for generative AI, predictive AI, and AI-powered DevEx solutions.
• Create automated test suites and evaluation pipelines for functional, regression, performance, and scenario-based testing.
• Analyze AI-generated outputs for defects, hallucinations, bias, inconsistency, reliability, and performance issues.
• Build test datasets, benchmark scenarios, prompt libraries, and evaluation artifacts.
• Define and report AI quality metrics, support root-cause analysis, and document test strategies and results.
• Collaborate with software, AI, and platform engineering teams on AI governance, responsible AI, and DevEx initiatives.

Requirements

• Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field.
• 3+ years of experience in software testing, quality engineering, test automation, or AI/ML testing.
• Strong Python programming and scripting skills, with experience building automated test frameworks and pipelines.
• Experience with APIs, automated integration testing, CI/CD environments, data validation, and defect investigation.
• Understanding of machine learning, generative AI, large language models, or AI-powered applications.
• Strong analytical, troubleshooting, communication, and collaborative skills in an agile environment.

Nice to have

• Experience testing generative AI, LLM, or Retrieval-Augmented Generation systems.
• Experience with DeepEval, Ragas, LangSmith, Azure AI, GitHub Copilot, OpenAI, Hugging Face, or LangChain.
• Knowledge of LLM observability, monitoring, traceability, prompt engineering, and benchmark dataset creation.
• Experience with performance, scalability, reliability, Responsible AI, or AI governance practices.

Culture & Benefits

• Contract engagement with a 40-hour work schedule.
• Collaborative work with software engineers, AI engineers, and platform teams.
• Eligible employees may receive medical, dental, vision, spending account, life insurance, and voluntary plan options.
• Eligible employees may participate in a 401(k) plan.