AI SDET (Software Development Engineer in Test)

TL;DR

AI SDET (Software Development Engineer in Test) (Python/AI Evaluation): Building and operating automated evaluation systems for AI agents with an accent on non-deterministic output testing, regulatory quality gates, and statistical validation. Focus on designing adversarial scenarios, validating shadow-mode performance against human benchmarks, and ensuring AI outputs remain safe, compliant, accurate, and reliable.

Location: Remote from Brazil

Company

CI&T delivers AI deployment, application modernization, data and technology-integrated business solutions for large enterprises across more than 25 countries.

What you will do

• Own and manage the Langfuse evaluation harness, including Golden Paths and adversarial test scenarios.
• Validate regulatory quality gates, including SR 26-2 and NYDFS Regulation 187.
• Run shadow-mode validation to compare AI performance with human benchmarks.
• Test data quality degradation, population segmentation errors, and human-in-the-loop checkpoint reliability.
• Verify that explanation blocks and signal types remain compliant with Designer output standards.
• Collaborate with engineers to identify and fix logic failures.

Requirements

• Advanced proficiency in Python and automated testing frameworks, including Pytest.
• Direct experience with AI evaluation tools such as Langfuse, RAGAS, or similar platforms.
• Experience with LLM-as-a-judge evaluation patterns.
• Knowledge of model risk management and regulatory compliance testing.
• Experience with shadow-mode deployment strategies and statistical validation.
• A skeptical, adversarial mindset and uncompromising attention to regulatory evidence.

Culture & Benefits

• Remote home-office work in Brazil.
• Collaborative work with engineers and specialists across an international organization.
• Benefits are provided according to CI&T's Brazil program.
• An assessment is included in the selection process, with support available for obtaining it if needed.