Manager; AI Evaluation Engineering

Location: Chicago, Illinois, United States

Salary: $147,760–$240,110 per year

Company

hirify.global builds products, services, and digital solutions that support customers and dealers and contribute to stronger, more sustainable communities.

What you will do

• Lead a team responsible for evaluating and validating generative AI solutions, including intelligent agents and digital assistants.
• Provide technical direction and support while aligning AI evaluation work with company goals.
• Oversee team and individual performance and identify training and development needs.
• Own the quality of AI engineering products, ensuring robust, reliable solutions.
• Establish and supervise engineering best practices across development and evaluation processes.

Requirements

• Experience leading software quality, test automation, validation, and AI evaluation teams.
• Experience delivering enterprise-scale software and generative AI solutions across hybrid cloud and embedded or edge environments.
• Expertise in GenAI architecture, LLMs, SLMs, multimodal and real-time models, prompt engineering, agentic systems, tool orchestration, RAG, vector databases, embeddings, chunking, and fine-tuning.
• Strong understanding of AI evaluation for non-deterministic systems, including metric design, RAG quality, reliability, safety evaluation, A/B testing, and human-in-the-loop validation.
• Experience with AI development, evaluation, observability, and monitoring technologies such as Azure, AWS, GCP, LangChain, LangGraph, Langfuse, Arize, LangSmith, Ragas, DeepEval, and Phoenix.
• Visa sponsorship is not available for this position.

Culture & Benefits

• Medical, dental, and vision benefits.
• Paid time off, including vacation, holidays, and volunteer time.
• 401(k) savings plans, HSA, and FSA options.
• Career development and tuition reimbursement.
• Parental leave, adoption benefits, life insurance, disability benefits, and employee assistance programs.
• Incentive bonus and employee discounts.