Назад
14 часов назад

Certification Assessment Specialist (AI)

207 000 - 230 000$
Формат работы
hybrid
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Certification Assessment Specialist (AI): Designing, building, testing, launching, and maintaining hands-on technical certification assessments with an accent on realistic scenarios, assessment environments, evidence capture, and traceable scoring. Focus on model-assisted scoring validation, end-to-end testing, delivery integrations, and maintaining reliable assessment workflows as tools and products change.

Location: San Francisco, United States; hybrid

Salary: $207K–$230K annually, plus equity

Company

OpenAI is an AI research and deployment company developing and safely deploying general-purpose AI products.

What you will do

  • Translate assessment blueprints and rubrics into realistic technical scenarios, candidate instructions, hands-on tasks, and evidence requirements.
  • Define, build, and configure assessment environments with repeatable provisioning and reset workflows for accounts, tools, data, and permissions.
  • Build pipelines that capture submissions and scoring evidence, apply scoring rules, route results for review, and protect sensitive data.
  • Build and validate model-assisted scoring against calibrated human judgments, investigate disagreements, and define human-review routing rules.
  • Test assessments end to end through dry runs and pilots, including accessibility, timing, failure recovery, multiple valid approaches, partial submissions, and platform failures.
  • Prepare launches, maintain versioned tasks and environments, support delivery integrations, investigate defects, and use feedback and scoring data to improve or retire assessments.

Requirements

  • Hands-on experience designing and building performance-based technical assessments, labs, or simulations for roles such as software developers or solution architects, including testing, launch, and maintenance; question-and-answer tests alone do not meet this requirement.
  • Experience using scripting and automation to build and troubleshoot technical assessment environments and translate realistic solutions with technical subject matter experts.
  • Experience in a technology company or similarly fast-paced environment with changing requirements and evolving business needs.
  • Ability to use blueprints and rubrics to design realistic tasks and define evidence that distinguishes competent performance from task completion.
  • Experience building or maintaining workflows that capture submissions, process evidence, and produce traceable scores or reviewer outputs.
  • Experience implementing or validating model-assisted scoring against expert judgments and determining when human review is required.
  • Experience working with external technical experts or delivery providers and testing integrations end to end.
  • Experience using pilot results, candidate feedback, and scoring or delivery data to improve assessment quality.
  • Understanding of accessibility requirements and protection of candidate evidence and confidential assessment content; clear communication of technical tradeoffs across technical and operations stakeholders.

Nice to have

  • Experience building hands-on assessments for AI application development, agentic workflows, or customer deployments.
  • Experience working with psychometricians or measurement specialists on rubric calibration, scoring consistency, or performance-assessment validation.
  • Experience launching performance-based certification assessments across live and remote delivery.

Culture & Benefits

  • Full-time individual contributor role reporting to the Technical Certification Lead.
  • Collaboration with certification, measurement, engineering, operations, subject matter experts, and external delivery providers.
  • Work spans prototype, pilot, launch, and ongoing maintenance across live and virtual delivery.
  • Equity is included in the compensation package.
  • OpenAI provides equal employment opportunity and reasonable accommodations; background checks follow applicable law.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →