Чётко очерченная роль в ведущей AI-компании сочетает практическую разработку оценок с использованием моделей и работу в быстрорастущей сфере; главный нюанс - требование узкого профильного опыта.
?Узкий профиль
Сфера AI
Понятная роль
Сильная компания
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Подключить Plus
Описание вакансии
Текст:
Hirify AI/Оригинал
TL;DR
Certification Assessment Specialist (AI): Designing, building, testing, launching, and maintaining hands-on technical certification assessments with an accent on realistic scenarios, assessment environments, evidence capture, and traceable scoring. Focus on model-assisted scoring validation, end-to-end testing, delivery integrations, and maintaining reliable assessment workflows as tools and products change.
Location: San Francisco, United States; hybrid
Salary: $207K–$230K annually, plus equity
Company
OpenAI is an AI research and deployment company developing and safely deploying general-purpose AI products.
What you will do
Translate assessment blueprints and rubrics into realistic technical scenarios, candidate instructions, hands-on tasks, and evidence requirements.
Define, build, and configure assessment environments with repeatable provisioning and reset workflows for accounts, tools, data, and permissions.
Build pipelines that capture submissions and scoring evidence, apply scoring rules, route results for review, and protect sensitive data.
Build and validate model-assisted scoring against calibrated human judgments, investigate disagreements, and define human-review routing rules.
Test assessments end to end through dry runs and pilots, including accessibility, timing, failure recovery, multiple valid approaches, partial submissions, and platform failures.
Prepare launches, maintain versioned tasks and environments, support delivery integrations, investigate defects, and use feedback and scoring data to improve or retire assessments.
Requirements
Hands-on experience designing and building performance-based technical assessments, labs, or simulations for roles such as software developers or solution architects, including testing, launch, and maintenance; question-and-answer tests alone do not meet this requirement.
Experience using scripting and automation to build and troubleshoot technical assessment environments and translate realistic solutions with technical subject matter experts.
Experience in a technology company or similarly fast-paced environment with changing requirements and evolving business needs.
Ability to use blueprints and rubrics to design realistic tasks and define evidence that distinguishes competent performance from task completion.
Experience building or maintaining workflows that capture submissions, process evidence, and produce traceable scores or reviewer outputs.
Experience implementing or validating model-assisted scoring against expert judgments and determining when human review is required.
Experience working with external technical experts or delivery providers and testing integrations end to end.
Experience using pilot results, candidate feedback, and scoring or delivery data to improve assessment quality.
Understanding of accessibility requirements and protection of candidate evidence and confidential assessment content; clear communication of technical tradeoffs across technical and operations stakeholders.
Nice to have
Experience building hands-on assessments for AI application development, agentic workflows, or customer deployments.
Experience working with psychometricians or measurement specialists on rubric calibration, scoring consistency, or performance-assessment validation.
Experience launching performance-based certification assessments across live and remote delivery.
Culture & Benefits
Full-time individual contributor role reporting to the Technical Certification Lead.
Collaboration with certification, measurement, engineering, operations, subject matter experts, and external delivery providers.
Work spans prototype, pilot, launch, and ongoing maintenance across live and virtual delivery.
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →