Software Development Engineer In Test (ML/AI)

TL;DR

Software Development Engineer in Test (ML/AI): Building automation frameworks and quality strategies for AI-powered workflows and ML services with an accent on model evaluation and LLM-assisted test generation. Focus on building scalable validation for LLM behavior, reducing flakiness in non-deterministic systems, and integrating AI testing into CI/CD pipelines.

Location: Hybrid (Must be based in the US)

Salary: $140,500 — $210,700 USD

Company

Sony Interactive Entertainment (PlayStation) is a global leader in interactive entertainment, producing iconic gaming consoles and software.

What you will do

• Define quality strategies, test plans, and automation coverage for ML-powered services and platform components.
• Use LLMs and AI-assisted techniques to generate and maintain high-value test cases for AI workflows.
• Design scenario-based test suites including adversarial prompts, edge cases, and ambiguous inputs.
• Develop scalable automation frameworks for backend services and ML inference systems using Python and/or Java.
• Build automated validation for LLM outputs, focusing on hallucination indicators and probabilistic model evaluation.
• Partner with engineering and ML teams to integrate automated testing into CI/CD pipelines.

Requirements

• 3+ years of experience as an SDET or QE engineer focused on backend and distributed systems.
• Experience using LLMs to generate and prioritize test cases for AI-powered experiences.
• Proficiency in Python, Java, or JS for automation development.
• Strong experience testing RESTful APIs and microservices architectures.
• Hands-on experience with automation frameworks such as pytest, JUnit, Selenium, Playwright, Cypress, or Appium.
• Experience with cloud/container technologies (AWS, GCP, Kubernetes, Docker).

Nice to have

• Experience validating ML outputs using statistical analysis.
• Familiarity with ML infrastructure (Seldon, KServe, Ray Serve) or Databricks.
• Prior work in content moderation ML, security, or fraud detection.
• Experience testing high-scale, low-latency online services.

Culture & Benefits

• Comprehensive medical, dental, and vision insurance.
• Matching 401(k) plan.
• Paid time off and wellness programs.
• Employee discounts for Sony products.
• Hybrid working policy.