Senior Software Engineer (AI Inference)

Location: Hybrid in Sunnyvale, CA, United States

Company

hirify.global builds AI accelerator hardware and inference systems based on its Wafer-Scale Engine for high-speed model training and inference.

What you will do

• Design, implement, test, operate, and evolve inference API features for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, multimodal inputs, and emerging inference capabilities.
• Lead multi-component delivery by writing design documents, evaluating tradeoffs, defining milestones, and coordinating implementation with engineers and partner teams.
• Implement consistent request and response behavior across GPU prefill, hirify.global decode, and other inference backends, including API validation, versioning, compatibility, and deprecation.
• Integrate foundation models, tokenizers, chat templates, generation configuration, sampling methods, serving frameworks, runtimes, and model-specific capabilities.
• Build routing, state-transfer integration, error handling, retries, cancellation, request lifecycle management, conformance tests, model-validation suites, benchmarks, metrics, tracing, logging, alerts, and diagnostic tools.
• Improve time to first token, streaming latency, throughput, resource efficiency, production resilience, documentation, debugging workflows, and customer and developer experience; mentor engineers and contribute to design and code reviews.

Requirements

• 5+ years of software engineering experience, including independent ownership of production services, distributed systems, or substantial software components.
• Strong Python programming ability and proficiency in Go, C++, Rust, or another language used to build production services.
• Experience designing and delivering multi-component features, documenting designs, evaluating tradeoffs, and coordinating implementation.
• Experience building and evolving production APIs with validation, error handling, compatibility, and testing practices; familiarity with REST, gRPC, or streaming protocols.
• Experience integrating software across service, framework, runtime, or infrastructure boundaries.
• Experience operating Linux services using containers, Kubernetes or comparable orchestration systems, and CI/CD.
• Ability to diagnose correctness, reliability, and performance issues across distributed components and deliver measurable improvements.
• Strong communication and collaboration skills, including leading technical delivery for a small project and supporting engineers through reviews or mentorship.
• Ability to turn ambiguous requirements into maintainable software while balancing customer needs, delivery timelines, and operational quality.
• Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience.

Nice to have

• Experience building or operating production ML or LLM inference APIs, including OpenAI-compatible interfaces.
• Experience extending or contributing to vLLM, SGLang, PyTorch, Hugging Face Transformers, Triton, TensorRT-LLM, or another ML systems project.
• Understanding of tokenization, chat templates, generation configuration, logits processing, stopping criteria, tool calling, structured generation, or constrained decoding.
• Experience with disaggregated prefill/decode architectures, KV-cache transfer, prefix caching, chunked prefill, or inference request scheduling.
• Experience building API conformance, model-quality, numerical-comparison, determinism, or performance-regression test systems.
• Experience with multi-model or multi-tenant serving, including routing, admission control, quotas, rate limiting, or capacity-aware scheduling.
• Experience developing performance-sensitive or highly concurrent services in Go, C++, Rust, or a similar language.
• Familiarity with GPU inference, AMD Instinct accelerators, ROCm, CUDA, collective communication, or multi-node model serving.
• Familiarity with reduced-precision inference and quantization formats such as BF16, FP8, FP4, INT8, or INT4.
• Experience building SDKs, developer tools, configuration systems, or self-service ML workflows, or collaborating directly with customers and open-source communities.

Culture & Benefits

• Build an AI platform designed to go beyond GPU constraints.
• Opportunities to publish and open-source AI research.
• Work with a high-performance AI supercomputer and a rapidly growing platform.
• Combination of job stability and startup vitality.
• Simple, non-corporate work culture that respects individual beliefs, with continuous learning, growth, and support.