JOB-1513
An AI company that serves its own models to customers worldwide wants an infrastructure engineer to operate its live model-serving stack. The job sits where cluster operations, GPU hardware and real-time performance meet, and it suits someone who likes tracing problems down through the whole stack.
Day to day
- Design, operate and expand resilient Kubernetes environments spanning AWS, GCP and specialist GPU hosts to meet worldwide demand for model inference
- Own the live serving path and diagnose faults across the NVIDIA stack, from drivers and CUDA through NCCL, plus node health and the fabric linking machines
- Partner with the team that optimizes models so low-level speedups hold up under real traffic and real hardware
- Evolve the release tooling so a single workflow ships models onto NVIDIA hardware, AWS Trainium and Google TPUs consistently
- Build monitoring that tracks first-token delay, latency between tokens, throughput and uptime, split by model, chip and region
- Lead reliability work: alerts, automatic failover, self-repairing systems and smart routing of traffic
What you bring
- Five or more years in infrastructure engineering, currently operating at staff level or above
- Current hands-on experience with inference infrastructure and LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM or Dynamo
- Running Kubernetes in production at large scale, across more than one cloud
- Strong GPU and accelerator knowledge, plus HPC networking such as InfiniBand, RDMA/EFA or MPI
- Observability and reliability practice for latency-sensitive services
- Bonus: heterogeneous accelerator fleets or open-source contributions to inference projects
On-site in the San Francisco Bay Area. Visa transfers considered.
