GPUs on Kubernetes: GPU Operator, Device Plugin, MIG & Time-slicing

Running GPUs under Kubernetes: what each GPU Operator component does, how pods get nvidia.com/gpu, MIG partitions on A100, H100, H200 and B200, time-slicing and MPS, and upgrading drivers on a live cluster.

Overview

Why this topic

Out of the box, Kubernetes knows nothing about GPUs: a node needs a kernel driver, a container runtime hook that puts the GPU into containers, and a device plugin that tells the kubelet how many GPUs it has. Monitoring, partitioning and validation come on top. The NVIDIA GPU Operator installs and runs all of it as Kubernetes workloads, so a GPU node is configured the same way as everything else in the cluster.

> [!IMPORTANT] Most GPU-on-Kubernetes incidents are one layer of this stack not being ready: no driver, no runtime hook, no device plugin. Know the order and you know where to look.

Interview questions

  • Pods requesting nvidia.com/gpu are stuck in Pending on a node with 8 GPUs. Debug it.
  • Explain MIG on A100, H100 and newer GPUs, and how you configure it in Kubernetes.
  • Time-slicing vs MPS vs MIG: when do you use each, and what can go wrong?
  • How do you upgrade the GPU driver on a live cluster without killing training jobs?
  • How do you schedule GPU workloads well: labels, taints, topology and fairness?
  • Do you run the driver in a container or on the host? Defend the choice for an on-prem fleet.

Sign up free to read the full topic

More AI Infrastructure topics

Start free on Cariara · Coding problems · System design studio