Autoscaling LLM Inference on Kubernetes

Scaling self-hosted LLM servers on Kubernetes: why CPU and request rate mislead, scaling on queue depth, KV-cache use and TTFT with KEDA or the HPA, cold start, scale-to-zero, and KServe, llm-d and the Gateway API Inference Extension.

Overview

Why the usual signals fail

A web service scales well on CPU or requests per second because each request costs roughly the same and the CPU is the bottleneck. An LLM server breaks both assumptions.

  • CPU utilization — The work runs on the GPU; the CPU sits mostly idle while the GPU is saturated
  • GPU utilization — A continuous-batching server keeps the GPU busy at moderate load too, so the number is high long before users wait
  • Requests per second — One request may be 50 tokens in and 10 out, the next 8,000 in and 1,000 out; the same RPS can mean idle or overloaded
  • Memory used — Servers such as vLLM reserve most GPU memory for the KV cache at startup, so it reads near full all the time

What users feel is time to first token (TTFT) and time per output token. Those rise when requests wait in the server's queue or when the KV cache is full and running requests are preempted. Scale on the signals that move before the latency does.

Interview questions

  • Your vLLM Deployment has an HPA on 70 percent CPU. Under a traffic spike, TTFT went from 400 ms to 20 s and the HPA never scaled. Explain, and fix it.
  • How do you choose between queue depth, KV-cache use and TTFT as the scaling metric?
  • A new replica of a 70B model takes 9 minutes to become ready. Break the time down and cut it.
  • Should a team scale its 30 fine-tuned model variants to zero when idle?
  • What do KServe, the Gateway API Inference Extension and llm-d each solve, and how do they relate?
  • After enabling autoscaling, replicas flap between 3 and 6 every few minutes and users see bursts of slow answers. Diagnose and fix.

Sign up free to read the full topic

More Agentic AI topics

Start free on Cariara · Coding problems · System design studio