Senior Testing Engineer (AI)

TL;DR

Senior Testing Engineer (AI): Designing scalable automation frameworks and validating AI/ML systems integrated into enterprise operations with an accent on UI, API, integration, and model-quality testing. Focus on building AI-specific test strategies, detecting LLM hallucinations, monitoring model drift, and driving quality gates through CI/CD.

Location: Remote-friendly role based in Bogota, Colombia.

Company

Robots & Pencils is an applied AI engineering firm that designs and delivers AI co-workers and enterprise software for clients across multiple industries.

What you will do

• Design and maintain scalable automation frameworks for UI, API, and integration testing.
• Integrate automated tests into CI/CD pipelines and establish quality gates.
• Develop AI-specific testing strategies, evaluate ML models, and validate LLM responses for hallucinations and consistency.
• Test data pipelines and feature engineering processes while monitoring model quality in production.
• Lead test planning, automation code reviews, release validation, and quality improvement initiatives.
• Collaborate with developers, ML engineers, DevOps, and product teams, while beginning to mentor junior engineers.

Requirements

• 3–4+ years of experience in QA, data QA, or AI testing.
• Strong Python programming skills plus at least one additional language such as Java or JavaScript.
• Hands-on experience with automation frameworks including Selenium, Cypress, Playwright, PyTest, TestNG, or Cucumber.
• Knowledge of the ML lifecycle, model evaluation metrics, LLM validation, hallucination testing, and data pipeline validation.
• Experience with CI/CD, API automation, version control, MLOps pipelines, and production model monitoring.
• Understanding of Agile methodologies and test management tools such as Jira or TestRail.

Nice to have

• Mobile automation, performance testing, LLM testing frameworks, prompt engineering, or cloud ML platform experience.
• Experience with Appium, JMeter, AWS SageMaker, or Azure ML.

Culture & Benefits

• Work on production-ready AI systems designed for real human workflows and measurable client impact.
• Collaborate with highly experienced teams averaging more than 15 years of experience.
• Operate in a fast-moving environment that values ownership, direct communication, craftsmanship, and continuous learning.
• Use modern AI assistants such as Claude and Cursor to improve delivery quality and speed.