Назад
3 дня назад

Software Engineer, Training Infrastructure (AI)

300 000 - 400 000$
Формат работы
onsite
Тип работы
fulltime
Грейд
senior
Английский
b2
Страна
US
Вакансия из списка Hirify.GlobalВакансия из Hirify Global, списка международных tech-компаний
Для мэтча и отклика нужен Plus

Мэтч & Сопровод

Для мэтча с этой вакансией нужен Plus

Описание вакансии

Текст:
/
TL;DR
Software Engineer, Training Infrastructure (AI): Building and operating distributed infrastructure for frontier model training and Tinker fine-tuning across large GPU clusters with an accent on scheduling, fault tolerance, throughput, and cost efficiency. Focus on optimizing large-scale model architectures, supporting LoRA and reinforcement learning workloads, and designing recovery and observability systems that withstand hardware and network failures.

Location: Onsite in San Francisco, California or New York, New York, United States

Annual salary: $300,000–$400,000 USD, depending on background, skills, and experience.

Company

Thinking Machines Lab builds AI systems that extend human will and judgment, including frontier models, the Tinker fine-tuning API, and human-AI interfaces.

What you will do

  • Design, build, and operate distributed training infrastructure for Tinker and internal frontier model training, including scheduling, orchestration, checkpointing, and recovery.
  • Improve training throughput and cost efficiency through profiling, parallelism strategies, memory optimization, and kernel- and communication-level tuning.
  • Bring up and optimize support for open-weights and in-house model architectures, including mixture-of-experts models.
  • Build reliability through automated failure detection, job recovery, multi-tenant isolation, and observability for long-running jobs.
  • Support multi-tenant LoRA fine-tuning and tightly coupled sampling and training for reinforcement learning workloads.
  • Collaborate with API, frameworks, and product teams to keep infrastructure and the user-facing SDK consistent and reliable.

Requirements

  • 4+ years of software engineering experience, including building and operating production distributed systems.
  • Strong Python programming skills plus experience with a systems language such as C++, Rust, or Go.
  • Hands-on experience with large-scale model training or inference infrastructure using frameworks such as PyTorch.
  • Demonstrated ability to debug complex performance and reliability issues in distributed environments.

Nice to have

  • Experience training or serving large language models at scale, including mixture-of-experts architectures.
  • Experience with LoRA and other parameter-efficient fine-tuning methods, or supervised and reinforcement learning pipelines for LLMs.
  • Experience with cluster orchestration and scheduling, such as Kubernetes or Slurm, and fault-tolerant checkpointing.
  • Experience building observability and profiling tooling.
  • Contributions to open-source machine learning systems projects such as PyTorch, Megatron, vLLM, SGLang, or similar.

Culture & Benefits

  • Full-time role in the Core Engineering department.
  • Visa sponsorship is available, with support through the visa process when appropriate.
  • Health, dental, and vision benefits.
  • Unlimited paid time off and paid parental leave.
  • Relocation support as needed.

Будьте осторожны

Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →