Назад
Company hidden
7 дней назад

Staff Applied Scientist (AI)

200 000 - 300 000$
Формат работы
remote (только USA)
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Staff Applied Scientist (AI): Building an evaluation framework for enterprise planner-based agents with an accent on user simulation, controlled experimentation, statistical grading, and cost-quality analysis. Focus on designing production-grade research tooling, comparing models and optimization levers, and delivering dashboard-backed recommendations with measurable dollar impact.

Location: US Remote

Salary: $200,000–$300,000 per year

Company

Robots & Pencils designs AI systems and agent solutions for real-world workflows, combining engineering with creative problem-solving.

What you will do

  • Design and build an end-to-end evaluation framework for planner-executor agents, including scenario schemas, user simulation, execution, grading, comparison, and automated insights.
  • Build realistic multi-turn LLM user simulation, isolated parallel experiment execution, gateway-based model routing, trace attribution, and optimization-lever plug-ins.
  • Develop statistical grading and comparison methods covering verified success, cost per success, p95 latency, confidence intervals, paired tests, and non-inferiority testing.
  • Convert experimental results into projected production savings, cost-quality frontiers, ranked evidence-backed insights, and generalization analysis.
  • Build a locally hosted, API-first dashboard and run the production baseline, model screen, lever screen, and confirmation campaign.
  • Partner with client engineers and leadership, deliver technical and executive reports, set experimental standards, and mentor engineers on rigorous agent evaluation.

Requirements

  • Experience with LLM and agent evaluation methods, including state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring, and user simulation.
  • Production experience building or evaluating planner-executor or other multi-step, tool-using agent architectures.
  • Strong experimental design and statistical inference background, including paired designs, non-inferiority testing, power analysis, and identifying inconclusive results.
  • Experience with multi-objective optimization or cost-aware machine learning, Pareto frontier analysis, and projected dollar-savings calculations.
  • Familiarity with LLM routing, model cascades, model selection, inference internals, prompt and plan caching, context compaction, reasoning budgets, and cache-aware model switching.
  • Experience with LLM observability and tracing, including request-level token, cache-usage, latency, and cost attribution.
  • Ability to build production-quality research tooling with Python services, containerized and sandboxed execution, and data pipelines.
  • Prior customer-facing or consulting experience delivering scoped research engagements.
  • Ability to explain technical findings to non-technical audiences, translate them into business impact, and provide deeper technical reasoning when challenged.

Nice to have

  • Publications or open-source contributions in LLM evaluation, agents, or efficient inference.
  • Experience building internal dashboards or web UIs that client teams can adopt.
  • Hands-on experience with LLM gateways or tracing stacks such as LangFuse or LangSmith.
  • Experience in regulated or high-stakes enterprise domains.
  • Production experience analyzing cost-quality tradeoffs.

Culture & Benefits

  • Research-heavy, customer-facing work with significant production tooling ownership.
  • Engineering is paired with creativity, with emphasis on ownership, craft, and accountable outcomes.
  • Work directly with client engineering teams and business stakeholders.
  • AI-assisted development tools such as Claude Code are part of the engineering context.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →