Senior AI Evaluation Engineer (CI/CD & Platform)

Location: Dublin, Ireland. Flex work requires spending at least 50% of each quarter in the office or in the field with customers, prospects, and partners.

Base salary: €96,000–€144,000 EUR per year.

Company

hirify.global is a Fortune 500 company and AI platform for managing people, money, and agents.

What you will do

• Build tiered continuous evaluation pipelines for static, schema, synthetic, multi-turn trajectory, and LLM-as-a-Judge evaluations.
• Architect and maintain CI/CD pipelines, skill registry versioning, and runner environments with automated release gates.
• Develop synthetic data generation and automated red-teaming utilities from API schemas and domain specifications.
• Curate versioned golden datasets and run regression tests across model upgrades, prompt changes, and security boundaries.
• Ingest production traces and failure logs into benchmark datasets to close test coverage gaps.

Requirements

• 4+ years of professional software engineering experience building production-grade systems in Python, with hands-on strength in either CI/CD pipeline engineering or AI evaluation methodology and working familiarity with the other.
• 2+ years designing, building, and operating test automation frameworks such as pytest, CI/CD build pipelines using GitHub Actions or GitLab CI, or automated release gates.
• 1+ years of hands-on experience evaluating or building Generative AI applications, including LLM tool-calling, agent workflows, or structured output parsing.
• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• Experience with AI evaluation and observability frameworks such as LangSmith, Braintrust, DeepEval, Promptfoo, or Phoenix.
• Strong knowledge of JSON Schema, Pydantic, OpenAPI, and API contract testing.
• Familiarity with MCP, tool-calling specifications, or agent execution harnesses.
• Experience with schema validation frameworks, linters, and test automation tooling; understanding of hallucinations, prompt injection, PII/PHI sanitization, and RBAC boundaries.
• Engineering rigor around test coverage, dataset versioning, evaluator calibration metrics, and regression prevention, plus the ability to own domains end-to-end, collaborate with platform and product teams, and write technical documentation and architecture designs.

Culture & Benefits

• Flexible work combines in-person and remote work, with at least half of each quarter spent in the office or field depending on the role.
• Eligible employees may receive a hirify.global Bonus Plan or role-specific commission/bonus and annual refresh stock grants.
• hirify.global provides reasonable accommodations during the employment application process.
• hirify.global is an equal opportunity employer and considers qualified applicants with arrest and conviction records in accordance with applicable Fair Chance law.