Назад
обновлено 2 месяца назад

Tech Lead Machine Learning Infrastructure Engineer (Recommendations and Search)

187 040 - 438 000$
Формат работы
onsite
Тип работы
fulltime
Грейд
lead
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Tech Lead Machine Learning Infrastructure Engineer (Recommendations and Search) (Machine Learning/Distributed Systems): Building large-scale distributed training, inference, and serving platforms for TikTok recommendation and search engines with an accent on GPU clusters, 3D parallelism, and low-latency model serving. Focus on designing fault-tolerant multi-thousand-GPU systems, co-designing generative recommendation infrastructure, and enforcing security and tenant isolation at enterprise scale.

Location: San Jose, United States; fully in-person schedule up to 5 days a week

Salary: $187,040–$438,000 annually, with potential discretionary bonuses, incentives, and restricted stock units.

Company

USDS Joint Venture supports TikTok products through data privacy, cybersecurity, trust and safety, and protection of U.S. user data and systems.

What you will do

  • Drive the technical roadmap for distributed real-time machine learning training and inference platforms powering recommendation and search across multiple product areas.
  • Architect multi-node training systems using data, tensor, and pipeline parallelism, GPU clusters, orchestration pipelines, and SLI/SLO frameworks.
  • Build low-latency, high-throughput model serving infrastructure and optimize inference for massive live traffic.
  • Partner with applied ML researchers to develop production-ready generative recommendation systems and high-throughput multimodal data platforms.
  • Design automated fault detection, asynchronous checkpointing, and resilience mechanisms for large GPU clusters.
  • Implement encryption, access controls, tenant isolation, and other security and compliance controls across the compute stack.

Requirements

  • Bachelor’s or Master’s degree in Computer Science, Computer Engineering, or a related technical discipline.
  • 5+ years of software engineering experience with Python, C++ or Java, and large-scale distributed systems.
  • 3+ years of experience building and maintaining enterprise-scale machine learning infrastructure with GPU clusters, Kubernetes, or Slurm.
  • Deep knowledge of PyTorch or TensorFlow, GPU memory management, CUDA, and InfiniBand or RoCE networking.
  • Experience with production LLM training and inference, profiling, and removal of I/O, compute, or network bottlenecks.
  • Strong system troubleshooting, debugging, technical leadership, mentoring, and architecture communication skills.

Nice to have

  • Experience with Megatron, DeepSpeed, advanced communication-computation overlap, and zero-bubble pipeline scheduling.
  • Experience scaling LLM workloads across hundreds of GPUs, including KV cache management, prefill-decoding separation, and model quantization.
  • Experience with regulated industries, sovereign cloud, federal data security, CUDA, Triton, CUTLASS, TensorRT, or Triton Inference Server.
  • Knowledge of MLOps, mixture-of-experts routing infrastructure, specialized hardware optimization, and graph compilation.

Culture & Benefits

  • Fully in-person work supports speed, alignment, team development, and integrated execution.
  • Medical, dental, and vision insurance from day one, plus a 401(k) savings plan with company match.
  • Paid parental leave, short- and long-term disability coverage, life insurance, and wellbeing benefits.
  • 10 paid holidays, 10 paid sick days, and 17 days of paid personal time, with accrual increasing by tenure.
  • Inclusive workplace focused on creativity, curiosity, resilience, continuous learning, and collaboration.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →