AI Engineering Lead, Evals and Systems Integration

Location: Boston, Chicago, London, São Paulo, or Warsaw; office-based role.

Salary: $198,250–$237,500 annualized in Massachusetts; $189,500–$227,000 annualized in Illinois. Additional discretionary performance compensation may apply.

Company

Bain & Company is a global consulting firm building an internal product platform for agentic decisioning and applied AI.

What you will do

• Own the evaluation framework for the Growth AI Performance System, including quality standards, scoring rubrics, datasets, regression suites, and deployment-readiness thresholds.
• Build automated evaluation pipelines and CI/CD quality gates covering the data ontology layer, Agentic Core, retrieval pipelines, and product modules.
• Evaluate knowledge graph outputs, entity resolution, taxonomy consistency, API contracts, agentic orchestration, tool use, context passing, and end-to-end workflow reliability.
• Integrate evaluation infrastructure with security controls, audit logging, compliance requirements, and sensitive client-data handling.
• Coordinate evaluation across distributed engineering teams, identify root causes, and translate findings into prioritized engineering improvements.
• Engage with consulting teams and senior stakeholders after client deployments, presenting quality metrics and reliability assessments in business terms.

Requirements

• Deep experience designing and running evaluations for LLM-based and agentic AI systems across the full delivery lifecycle.
• 8 or more years in AI engineering, ML engineering, or applied AI roles with significant hands-on building responsibilities.
• Experience owning evaluation frameworks for production AI or ML systems, including infrastructure, standards, and deployment gates.
• Strong systems integration evaluation skills covering knowledge graphs, API contracts, data pipelines, and agentic context passing.
• Strong Python proficiency and experience building automated evaluation pipelines, regression suites, scoring rubrics, and CI/CD workflows.
• Statistical reasoning and experimental design skills for measuring quality, interpreting results, and identifying root causes.
• Understanding of ontology evaluation, entity resolution, taxonomy consistency, semantic-layer quality, human-in-the-loop annotation, preference learning, and feedback systems.
• Understanding of security, audit, compliance, responsible AI, bias and fairness evaluation, AI safety, prompt injection, data leakage, and agentic workflow exploitation.
• Active use of AI-assisted coding tools such as Cursor or Claude Code, strong troubleshooting and root-cause analysis skills, and experience in agile cross-functional product or consulting-adjacent environments.
• Associate’s or bachelor’s degree in computer science, engineering, mathematics, or a related technical field, or an equivalent combination of education, training, and experience.

Nice to have

• Experience working with or alongside knowledge graph, ontology, or semantic data teams.
• Direct experience engaging with clients or senior business stakeholders in a technical capacity.

Culture & Benefits

• Permanent full-time employment within a startup-style product team backed by Bain’s global scale and client relationships.
• Annual discretionary performance bonus and potential additional discretionary compensation.
• 4.5% 401(k) company contribution, increasing after three years and fully vested upon start.
• Company-paid individual medical, dental, and vision premiums, plus life and long-term disability insurance.
• Paid time off including parental leave, sick leave, and paid holidays.