Sr. Software Development Engineer in Test (AI)

TL;DR

Sr. Software Development Engineer in Test (AI): Owning test automation and quality frameworks for AI Voice Agent services with an accent on LLM evaluation and end-to-end product experiences. Focus on building evaluation frameworks to measure response quality, integrating AI quality checks into CI/CD, and mitigating risks such as hallucinations and bias.

Location: Must be based at one of Dialpad's Canadian offices (Kitchener/Ontario)

Salary: $135,500 - $158,000 CAD

Company

AI-native business communications platform unifying calling, messaging, and meetings powered by real-time AI.

What you will do

• Own end-to-end quality for agentic features, including strategy, development, and release qualification.
• Design and build automation tooling for AI/LLM-driven systems, including prompt flows and agent orchestration.
• Develop evaluation frameworks (evals) to measure response quality, accuracy, and hallucination rates.
• Integrate AI quality checks into CI/CD pipelines to ensure fast feedback cycles.
• Build tooling for LLM observability and debugging, including prompt tracing and response analysis.
• Mentor engineers and raise the bar on AI quality engineering practices across the organization.

Requirements

• 5+ years of experience in software engineering or SDET roles with a strong development focus.
• Proficiency in Python (preferred), Java, or JavaScript.
• Hands-on exposure to LLMs or AI/ML systems such as OpenAI, Claude, or Gemini.
• Experience testing distributed, cloud-native SaaS systems and APIs.
• Understanding of non-deterministic systems and probabilistic testing approaches.
• Must be based at one of Dialpad's Canadian offices.

Nice to have

• Experience with Go, Google Cloud Platform (GCP), Kubernetes, Redis, or ElasticSearch.
• Familiarity with Vue3 or React for frontend testing.
• Knowledge of prompt orchestration frameworks and evaluation tooling.

Culture & Benefits

• Work at the center of the AI transformation in business communications.
• Competitive salary and comprehensive benefits package.
• Access to cutting-edge AI tools and a robust professional training program.
• Inclusive and vibrant office environment designed to cultivate collaboration and connection.