AI Evaluation Specialist

Location: US; must be a U.S. Citizen or Lawful Permanent Resident and able to obtain a U.S. Government Public Trust 5 security clearance.

Company

hirify.global provides strategic consulting, program management, and data-driven analytics for federal public health agencies.

What you will do

• Plan and execute hands-on evaluations of enterprise AI tools, generative AI products, platforms, models, and agents using representative business scenarios, workflows, prompts, datasets, and user personas.
• Assess response quality, accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, performance, safety, privacy, security, bias, governance, and cost.
• Design and maintain evaluation frameworks, scorecards, rubrics, benchmarks, test cases, and measurable acceptance criteria.
• Conduct comparative evaluations and proof-of-concept assessments across AI products, models, configurations, vendors, integrations, RAG systems, enterprise search, connectors, APIs, and MCP-enabled capabilities.
• Analyze model and product behavior, identify failure modes, and translate findings into recommendations, metrics, risk findings, and regression reporting.
• Collaborate with product, data science, AI engineering, software engineering, security, privacy, legal, governance, and business stakeholders while contributing to knowledge sharing.

Requirements

• 5+ years of experience in model evaluation or model quality assurance, including hands-on evaluation of AI-enabled products or services.
• Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
• Hands-on experience with generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes.
• Experience assessing AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost.
• Familiarity with large language models, retrieval-augmented generation, AI agents, APIs, enterprise search, and tool or connector integrations.
• Bachelor's degree.
• Must be a U.S. Citizen or Lawful Permanent Resident (Green Card Holder).
• Must be a U.S. Citizen and able to obtain a U.S. Government security clearance (Public Trust 5) within a reasonable period.

Nice to have

• Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models such as Rosalind or comparable solutions.
• Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing.
• Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows.
• Experience developing dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results.
• Experience using Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings.
• Industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI.
• Master's degree preferred.

Culture & Benefits

• Market-competitive salary commensurate with experience and education.
• Medical, dental, vision, and life insurance.
• Paid time off and 401(k) with company match.
• Growth and promotion opportunities.
• Interview responses must be based on the candidate's own knowledge, skills, and experience; AI-assisted interview responses are not permitted and may lead to disqualification.