AI Test Automation Engineer (AI)

Location: Remote, Almaty

Salary: $4,000–$4,500 gross per month

Company

OnTarget Labs is an international software product development company creating next-generation product lines.

What you will do

• Write and maintain automated tests for backend services and APIs.
• Test AI-powered features and services, validating outputs and behavior.
• Build AI evaluations into CI/CD pipelines to track quality as models, prompts, and data evolve.
• Test asynchronous and event-driven workflows, data pipelines, and third-party AI integrations.
• Validate performance, scalability, reliability, quality gates, and test reporting.
• Collaborate with backend engineers and product stakeholders to define expected behavior.

Requirements

• 3+ years of backend test automation experience.
• Strong Python and JavaScript/TypeScript skills.
• Experience with pytest, unittest, or similar test frameworks.
• Experience testing APIs with Postman, Requests, HTTPX, or similar tools.
• Experience with Docker, SQL, cloud environments, and non-functional testing.
• Experience testing backend systems with AI, ML, or LLM components and using LLM evaluation tools such as DeepEval, Ragas, or LangSmith.

Nice to have

• Experience building or maintaining AI evaluations and testing non-deterministic systems.
• Contract testing experience with Pact or similar tools.
• Exposure to prompt engineering, RAG-based systems, observability tools, security testing, or synthetic test data.
• Fluent English.

Culture & Benefits

• Full-time remote work.
• Opportunities for professional growth.
• Business trips abroad may be available.