Lead Prompt Engineer (AI)

TL;DR

Lead Prompt Engineer (AI): Guiding a team through technical migration of templates to LLM autoraters with an accent on prompt optimization and model performance enhancement. Focus on designing complex manual prompts, analyzing error patterns, and ensuring model quality against human baselines.

Location: Must be based in the United States

Company

A global provider of AI services and data validation solutions.

What you will do

• Utilize Automated Prompt Generation tools for complex template clusters.
• Supervise Automated Prompt Optimization tools and refine outputs when performance plateaus.
• Manually draft and test prompts to navigate complex template architectures and edge cases.
• Monitor shadowbot runs to track human vs LLM rating disagreements.
• Calculate and track model accuracy metrics including F1 scores, precision, and recall.
• Draft technical launch readiness documentation for project finalization.

Requirements

• Native fluency in English required.
• Must be based in the United States and eligible to work according to Federal law.
• Master’s or Doctorate degree in Computer Science, Data Science, Computational Linguistics, or related field.
• At least 7 years of professional experience as a Prompt Engineer.
• Strong proficiency in identifying error patterns and analyzing model performance using SQL or data tools.
• Experience tuning Large Language Models for structured outputs and complex classification tasks.

Nice to have

• Familiarity with enterprise-grade LLM interfaces such as the Goose API.
• Hands-on experience with Automated Prompt Optimization systems.
• Expertise in semantics, logic, and computational linguistics.

Culture & Benefits

• Collaborative environment focused on shaping the future of AI-driven solutions.
• Opportunities to lead strategy and mentor junior engineering talent.
• Engagement in cutting-edge AI model evaluation and tuning workflows.