Чётко очерченная специализация на проверке ИИ и промышленной оценке моделей сочетается с серьёзной технической глубиной; главный компромисс - работа только в офисе.
Только офис
Чёткие задачи
Актуальная область
Независимая проверка
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Подключить Plus
Описание вакансии
Текст:
Hirify AI/Оригинал
TL;DR
AI Assurance Engineer - MLE Focus (AI/ML): Building evaluation infrastructure, benchmark datasets, and reproducible testing capabilities for recommender and generative AI systems with an accent on statistical rigor, behavioral diagnosis, and safety assurance. Focus on designing adversarial tests, detecting regressions and drift, and integrating metrics, replay workflows, and benchmarks into assurance CI/CD.
Location: San Jose R&D - Security; fully in-person schedule up to 5 days a week.
Salary: $187,040–$438,000 annually, plus potential discretionary bonuses, incentives, and restricted stock units.
Company
TikTok USDS Joint Venture operates data privacy, cybersecurity, trust and safety, and content assurance programs protecting U.S. user data, applications, and algorithms.
What you will do
Define reproducible evaluation protocols, benchmark suites, sampling strategies, datasets, statistical analyses, metrics, baselines, and acceptance criteria for recommender and generative AI systems.
Build and maintain benchmark and adversarial datasets, including harm-exposure samples, ranking-quality cases, jailbreak prompts, leakage tests, and counterfactual replay inputs.
Implement behavioral tests covering safety, ranking quality, fairness, consistency, refusal correctness, factuality within scope, provenance, adversarial robustness, drift, and regressions.
Diagnose model-behavior regressions and post-release drift while distinguishing model effects from data, sampling, or infrastructure issues.
Integrate benchmarks, metrics, replay workflows, red-team assets, telemetry, and failure-diagnosis capabilities with shared evaluation runtimes and assurance CI/CD.
Produce traceable technical findings while maintaining independence from the product teams whose models are evaluated.
Requirements
Bachelor’s degree or higher in Computer Science, Machine Learning, Statistics, or a related technical field, or equivalent practical experience.
Experience evaluating machine learning systems using clearly defined metrics, datasets, baselines, and reproducible test protocols.
Experience developing production-quality evaluation or data-processing software in Python, Go, C++, or a comparable language.
Strong knowledge of sampling, hypothesis testing, uncertainty, experiment design, error analysis, and regression diagnosis.
Nice to have
Experience with recommender-system evaluation, ranking metrics, counterfactual replay, harm-exposure sampling, cohort consistency, or ranking-quality benchmarks.
Experience evaluating generative AI refusal behavior, jailbreak resistance, leakage, factuality within scope, provenance, or multimodal systems.
Experience integrating benchmarks and metrics with test harnesses, replay pipelines, experiment systems, or CI/CD workflows.
Ability to communicate evidence, limitations, and technical findings clearly to product engineering, Trust & Safety, business owners, and governance partners while maintaining ownership boundaries.
Culture & Benefits
Fully in-person work up to 5 days per week as the company shifts from a hybrid model.
Medical, dental, and vision insurance from day one, plus a 401(k) savings plan with company match.
Paid parental leave, short- and long-term disability coverage, life insurance, and wellbeing benefits.
10 paid holidays, 10 paid sick days, and 17 days of Paid Personal Time, prorated upon hire with increasing accruals by tenure.
Potential discretionary bonuses, incentives, and restricted stock units; benefits may vary by employment type and country of work.
Inclusive, diverse workplace focused on curiosity, humility, resilience, continuous learning, and meaningful impact.
Будьте осторожны
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →