Вакансия из Hirify Global, списка международных tech-компаний Для мэтча и отклика нужен Plus
Пожаловаться
Зарплата и рынок
развернуть
$200к-300к/год
$200к$240кмедиана$290к
По похожим ролям (49) ~ $240к/год
На СНГ рынке: $4.4к/мес ($3.6к - $5.4к)
87
Отличная вакансия
развернуть
Чёткий фокус на оценке LLM-агентов в production сочетается с широкой технической и клиентской ответственностью, что может сделать нагрузку высокой в аутсорсинговой компании.
Широкая зона ответственности
?Клиентский формат
Понятные задачи
Передовой ИИ
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Подключить Plus
Описание вакансии
Текст:
Hirify AI/Оригинал
TL;DR
Staff Applied Scientist (AI): Building an evaluation framework for enterprise planner-based agents with an accent on user simulation, controlled experimentation, statistical grading, and cost-quality analysis. Focus on designing production-grade research tooling, comparing models and optimization levers, and delivering dashboard-backed recommendations with measurable dollar impact.
Location: US Remote
Salary: $200,000–$300,000 per year
Company
Robots & Pencils designs AI systems and agent solutions for real-world workflows, combining engineering with creative problem-solving.
What you will do
Design and build an end-to-end evaluation framework for planner-executor agents, including scenario schemas, user simulation, execution, grading, comparison, and automated insights.
Build realistic multi-turn LLM user simulation, isolated parallel experiment execution, gateway-based model routing, trace attribution, and optimization-lever plug-ins.
Develop statistical grading and comparison methods covering verified success, cost per success, p95 latency, confidence intervals, paired tests, and non-inferiority testing.
Convert experimental results into projected production savings, cost-quality frontiers, ranked evidence-backed insights, and generalization analysis.
Build a locally hosted, API-first dashboard and run the production baseline, model screen, lever screen, and confirmation campaign.
Partner with client engineers and leadership, deliver technical and executive reports, set experimental standards, and mentor engineers on rigorous agent evaluation.
Requirements
Experience with LLM and agent evaluation methods, including state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring, and user simulation.
Production experience building or evaluating planner-executor or other multi-step, tool-using agent architectures.
Strong experimental design and statistical inference background, including paired designs, non-inferiority testing, power analysis, and identifying inconclusive results.
Experience with multi-objective optimization or cost-aware machine learning, Pareto frontier analysis, and projected dollar-savings calculations.
Familiarity with LLM routing, model cascades, model selection, inference internals, prompt and plan caching, context compaction, reasoning budgets, and cache-aware model switching.
Experience with LLM observability and tracing, including request-level token, cache-usage, latency, and cost attribution.
Ability to build production-quality research tooling with Python services, containerized and sandboxed execution, and data pipelines.
Prior customer-facing or consulting experience delivering scoped research engagements.
Ability to explain technical findings to non-technical audiences, translate them into business impact, and provide deeper technical reasoning when challenged.
Nice to have
Publications or open-source contributions in LLM evaluation, agents, or efficient inference.
Experience building internal dashboards or web UIs that client teams can adopt.
Hands-on experience with LLM gateways or tracing stacks such as LangFuse or LangSmith.
Experience in regulated or high-stakes enterprise domains.
Production experience analyzing cost-quality tradeoffs.
Culture & Benefits
Research-heavy, customer-facing work with significant production tooling ownership.
Engineering is paired with creativity, with emphasis on ownership, craft, and accountable outcomes.
Work directly with client engineering teams and business stakeholders.
AI-assisted development tools such as Claude Code are part of the engineering context.
Будьте осторожны
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →