Senior Prompt Engineer (AI)

TL;DR

Senior Prompt Engineer (AI): Developing and optimizing complex LLM prompt templates and autorater workflows with an accent on prompt engineering techniques and automated prompt generation. Focus on navigating complex template architectures, tuning model performance for structured outputs, and maintaining high accuracy metrics against human-validated datasets.

Location: Must be based in Japan

Company

A global provider of translation, localization, and AI data services.

What you will do

• Create baseline prompts for parent-child template clusters using Automatic Prompt Generation (APG) tools.
• Supervise Automated Prompt Optimization (APO) tools, review outputs, and troubleshoot performance plateaus.
• Manually refine prompts to resolve complex architectural edge cases and anti-patterns.
• Monitor shadowbot runs and ensure appropriate discrepancy tracking between human and LLM ratings.
• Validate autorater quality against gold data by calculating accuracy metrics like F1 scores, precision, and recall.
• Draft technical launch readiness documentation for final project releases.

Requirements

• Language: Native fluency in Japanese and fluent in English required.
• Location: Must be currently based in Japan.
• Minimum 4 years of experience specifically as a Prompt Engineer.
• Degree in Computer Science, Data Science, Computational Linguistics, or related field.
• Proficiency in analyzing model performance patterns and using SQL or data analytics tools.
• Proven experience tuning LLMs for structured outputs, complex classification, and few-shot learning.

Nice to have

• Familiarity with enterprise LLM interfaces like the Goose API.
• Hands-on experience with Automated Prompt Optimization (APO) systems.
• Background in computational linguistics or software engineering.

Culture & Benefits

• Part-time remote work arrangement providing high flexibility.
• Opportunity to work on advanced AI model evaluation and tuning projects.
• Collaborative environment focused on technical innovation and data quality.