AI Automation Engineer

Location: Vienna, Virginia, United States; hybrid work with onsite and remote attendance

Company

hirify.global provides outsourced services and workforce solutions across North America.

What you will do

• Design, develop, and maintain testing and evaluation frameworks for generative AI, predictive AI, and AI-powered developer experience solutions.
• Create automated test suites and evaluation pipelines for model behavior, output quality, accuracy, reliability, and performance.
• Perform functional, regression, performance, and scenario-based testing of AI-enabled applications and services.
• Analyze AI-generated outputs for defects, hallucinations, bias, inconsistencies, and other quality issues.
• Build test datasets, benchmark scenarios, prompt libraries, and evaluation artifacts.
• Collaborate with software, AI, and platform engineers on quality metrics, root-cause analysis, documentation, and responsible AI evaluation.

Requirements

• Bachelor’s degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field.
• 3+ years of experience in software testing, quality engineering, test automation, or AI/ML testing.
• Strong Python programming and scripting skills, with experience building automated test frameworks and pipelines.
• Experience with APIs, automated integration testing, CI/CD environments, data validation, defect investigation, and source control.
• Understanding of machine learning, generative AI, large language models, AI-powered applications, statistical analysis, and evaluation techniques.
• Strong analytical, troubleshooting, communication, and collaboration skills in an agile environment.

Nice to have

• Experience with generative AI, LLM, or Retrieval-Augmented Generation systems and evaluation tools such as DeepEval, Ragas, or LangSmith.
• Knowledge of LLM observability, monitoring, traceability, prompt engineering, and benchmark dataset creation.
• Experience with Azure AI, GitHub Copilot, OpenAI, Hugging Face, LangChain, performance testing, or reliability engineering.
• Experience with developer platforms, DevEx initiatives, enterprise, or regulated technology environments.

Culture & Benefits

• Contract role with a 40-hour work week and a hybrid onsite/remote model.
• Collaborative work with software engineers, AI engineers, and platform teams.
• Eligible employees may receive medical, dental, vision, spending account, life insurance, voluntary plan, and 401(k) benefits.