Serving LLMs On-prem: Triton, NIM, vLLM & the Model Store
Running LLM inference on owned GPUs: choosing Triton, NIM or vLLM, a versioned model store, GPU sizing from memory math, packing, rollouts, air-gapped installs, quotas and 24x7 operations.
Overview
Why this topic
A team that owns its GPUs cannot call a hosted API when a model misbehaves. It picks the server, keeps the weights, sizes the GPUs, rolls out new versions and answers the page at 3 a.m. Most of the work is not the model: it is storage, memory math, change control and the gateway in front.
> [!IMPORTANT] On-prem serving fails in boring places: a weights copy that differs between nodes, a model that no longer fits after the context length was raised, a version pushed to every node at once. Each has a known control.
Interview questions
- Size the GPUs for Llama 3.1 70B serving 32 concurrent conversations of up to 8K tokens.
- How do you roll out a new model version without an outage or a silent quality drop?
- Design the model store and the air-gapped install for a site with no internet access.
- Many teams share the serving fleet. How do you route requests and enforce quotas?
- You have 16 H100s and nine models of different sizes. How do you pack them?
- What do you watch and alert on to run LLM serving 24x7?
Sign up free to read the full topic
More AI Infrastructure topics
- Post-silicon Validation & Bring-up Labs
- Power & Cooling for GPU Racks
- PyTorch torch.compile, FSDP2 & Training Optimization
- Quantization & Model Compression
- Rack-scale NVLink: Operating GB200/GB300 NVL72
- ROCm & the AMD GPU Stack
- Silicon Root of Trust & Key Provisioning in Manufacturing
- Slurm for GPU Clusters
- Speculative Decoding & Parallel Token Generation
- Storage for GPU Training: Parallel Filesystems & GPUDirect Storage
- Tape-out Release Engineering & Signoff Gates
- The Chip Design Flow as a CI Pipeline
Start free on Cariara · Coding problems · System design studio