Model Serving

Production model serving — low latency, autoscaling, multi-framework, GPU-aware. KServe (CNCF, K8s-native), BentoML (Python framework), Seldon Core, NVIDIA Triton (GPU-optimized), Ray Serve, ONNX Runtime, batch vs online, GPU sharing (MIG, MPS).

Overview

What production serving requires

Model serving takes a registered model artifact and exposes it as a reliable, low-latency HTTP or gRPC endpoint for real traffic. The naive version, a Flask app that loads a pickle at startup, works for one model at single-digit QPS.

  • Low, predictable latency — Meet p50 and p99 SLOs, with cold-start mitigation
  • Autoscaling — Horizontal pod scaling on request rate or GPU utilization; scale-to-zero for cost
  • Multi-framework — One platform for sklearn, XGBoost, PyTorch, TensorFlow, ONNX and LLMs
  • GPU efficiency — Batch inflight requests, share GPU memory across small models, partition hardware
  • Observability — Per-request latency, per-model throughput, prediction logging
  • Controlled rollout — Canary, shadow and A/B traffic management at the serving layer

KServe

KServe (formerly KFServing) is the dominant Kubernetes-native serving framework. One InferenceService custom resource declares the model URI, framework runtime, autoscaling and canary split. It is the default serving layer in Kubeflow and is CNCF-incubating.

BentoML

BentoML is Python-first: you write a class with decorated API endpoints and BentoML handles packaging and deployment. Yatai is its Kubernetes operator.

> [!TIP] BentoML fits Python-native teams who want to stay in Python and avoid YAML-heavy serving configuration.

NVIDIA Triton

Triton Inference Server is the GPU-optimized multi-framework runtime (TensorFlow, PyTorch, ONNX, TensorRT, OpenVINO and Python backends). It is the default for high-throughput GPU inference outside LLM workloads.

  • Dynamic batching: concurrent requests coalesce into one GPU forward pass.
  • Model concurrency: several model instances run inside one Triton process.
  • Ensemble pipelines: multi-step inference graphs.

GPU sharing primitives

Ordered by isolation quality:

  • MIG (Multi-Instance GPU) — Hardware partition — Full (memory and compute) — A100/H100 only
  • MPS (Multi-Process Service) — Software sharing — Partial (no memory isolation) — Any NVIDIA GPU
  • Time-slicing — Context switching — None — Any NVIDIA GPU

> [!WARNING] For multi-tenant serving, use MIG when the hardware supports it. Time-slicing causes unpredictable latency spikes from context switching and has no memory isolation.

Interview questions

  • Compare KServe, BentoML, and NVIDIA Triton. When would you choose each?
  • How would you set up canary and A/B deployments for a model serving system on Kubernetes?
  • What are the GPU sharing options for multi-tenant model serving and when does each apply?
  • How do you handle serving multiple model versions simultaneously and ensure graceful rollback?
  • Walk through the operational concerns for a model serving platform at 10,000 QPS.

Sign up free to read the full topic

More MLOps & LLMOps topics

Start free on Cariara · Coding problems · System design studio