Senior Testing Engineer (AI)

TL;DR

Senior Testing Engineer (Python/AWS): Designing and implementing a comprehensive test architecture for a cloud-native AI/ML platform with an accent on agentic AI pipelines and serverless infrastructure. Focus on building scalable test suites from the ground up, implementing quality gates in CI/CD, and validating LLM output consistency.

Location: Canada

Company

An applied AI engineering firm that designs and ships AI co-workers and business architecture for enterprise operations.

What you will do

• Design and implement a scalable testing architecture for a Python/AWS stack including Lambda, DynamoDB, SQS, and EventBridge.
• Write production-grade test code using PyTest and develop agentic evaluation and synthetic learner pipelines.
• Own and manage quality gates within CI/CD pipelines using GitHub Actions and modify production code to improve testability.
• Establish testing standards and patterns for the broader engineering team, providing mentorship to junior and mid-level engineers.
• Collaborate with product and engineering leadership to align test strategies with delivery goals and platform architecture.

Requirements

• 5+ years of professional software engineering experience with a strong focus on unit, integration, and E2E testing.
• Strong Python programming skills and deep expertise with PyTest and Python-native testing frameworks.
• Hands-on experience with AWS services (Lambda, DynamoDB, SQS, S3, EventBridge) and CDK.
• Experience building or validating agentic or LLM-based systems, focusing on output consistency and hallucination validation.
• Proven expertise in CI/CD, specifically owning quality gates in delivery pipelines like GitHub Actions.
• Must be based in Canada.

Nice to have

• Working knowledge of AI safety and responsible AI principles (prompt injection defenses, PII handling).
• Familiarity with DynamoDB single-table design and its specific testing challenges.
• Experience using AI-forward development tools such as Claude Code and Cursor.

Culture & Benefits

• High-seniority environment with teams averaging 15+ years of experience.
• Culture of extreme ownership, craft obsession, and direct, honest communication.
• Fast-paced atmosphere that values resourcefulness and the ability to learn rapidly in the evolving AI landscape.
• Focus on shipping exceptional, production-ready work rather than theoretical perfection.