AI Evaluation Specialist (Generative AI)

Location: Remote within the continental United States; position associated with Georgia, United States

Salary: USD $102,000–$120,000 per year

Company

Chenega Services & Federal Solutions, LLC, part of hirify.global, provides professional and federal solutions.

What you will do

• Plan and execute hands-on evaluations of enterprise AI tools, including OpenAI and life sciences models such as Rosalind, using business scenarios, workflows, prompts, datasets, and user personas.
• Assess response quality, accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, safety, privacy, security, performance, and cost.
• Design and maintain evaluation frameworks, scorecards, rubrics, benchmarks, test cases, and acceptance criteria for AI tools, models, agents, and AI-enabled applications.
• Conduct comparative evaluations and proof-of-concept assessments across AI products, models, configurations, and vendors.
• Evaluate RAG, enterprise search, file and data analysis, connectors, APIs, AI agents, and MCP-enabled capabilities.
• Collaborate with product, data science, engineering, security, privacy, legal, governance, and business stakeholders; communicate findings and recommendations.

Requirements

• 5+ years of experience in model evaluation or model quality assurance, including hands-on evaluation of AI-enabled products or services.
• Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
• Hands-on experience with generative AI tools and large language models, including prompt design, systematic testing, output analysis, hallucination identification, and failure-mode analysis.
• Experience assessing AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost.
• Familiarity with large language models, RAG, AI agents, APIs, enterprise search, and tool or connector integrations.
• Bachelor’s degree.
• Teleworking is limited to the continental United States.

Nice to have

• Experience evaluating enterprise AI platforms or assistants, including OpenAI products, Rosalind, or comparable generative AI solutions.
• Experience with automated LLM evaluation frameworks, model-as-a-judge methods, human-in-the-loop evaluation, red teaming, or adversarial testing.
• Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows.
• Experience developing dashboards or reporting mechanisms for evaluation metrics, quality trends, risk findings, and regression results.
• Ability to use Python, SQL, spreadsheets, notebooks, or analytics platforms to analyze datasets and evaluation results.
• Industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI.
• Master’s degree.

Culture & Benefits

• Regular full-time employment with a collaborative environment, management engagement, recognition, and promotion opportunities.
• Paid time off, 11 paid holidays, and applicable sick leave.
• Medical, dental, vision, prescription, disability, life, and employee assistance benefits.
• Flexible spending, health savings, health reimbursement, dependent care, and commuter accounts.
• 401(k)/401(a) retirement plans and voluntary insurance benefits.
• Remote work is permitted within the continental United States.

Hiring process

• The position is contingent on contract award, and details may change based on the final award.
• Final salary depends on skill set, qualifications, and approved funding.