Batch GPU Scheduling on Kubernetes: Kueue, Volcano & Gang Scheduling
Running a shared GPU cluster as a batch system on Kubernetes: partial-placement deadlock, gang scheduling, Kueue quotas and preemption, Volcano PodGroups, topology-aware placement, JobSet and idle GPU reclamation.
Overview
Why this topic
Kubernetes was built for services: many small, independent pods that can start one at a time. A distributed training job is the opposite. It needs, say, 16 pods of 8 GPUs each, all running at once, close to each other on the network, and it is useless until the last one starts.
On a shared cluster with several teams, three more questions appear: who gets the GPUs when demand exceeds supply, who may use another team's idle share, and how that share comes back. Kueue and Volcano are the two common answers, and an interviewer expects you to know why they exist and how their objects fit together.
Interview questions
- Two 64-GPU training jobs are submitted to a cluster with 96 free GPUs, and neither ever starts. Explain why, and how you would fix it permanently.
- Design GPU quotas for four teams sharing 512 H100s, so no GPU sits idle while someone is waiting, but each team keeps a guaranteed share.
- How do you make a 256-GPU job land on nodes that share a network block, and why does placement matter so much for training?
- Compare running distributed training as a plain Job, a JobSet and a PyTorchJob. What does each give you?
- Production inference and research training share one GPU cluster. How do you set priorities and preemption so inference is never starved, and training does not lose days of work?
- GPU allocation on the cluster is 95 percent, but average utilization is 30 percent. How do you find and reclaim the waste?
Sign up free to read the full topic
More AI Infrastructure topics
- AI Accelerator Software Stacks: TPU, Trainium, MTIA & Bring-up
- Bare-metal Provisioning & Fleet Firmware
- CUDA Kernel Profiling, Occupancy & the Roofline
- CUDA Memory Hierarchy & Access Patterns
- Disaggregated Prefill-Decode Architecture
- Distributed Training: NCCL, Collectives & Parallelism
- DPUs in GPU Clusters: BlueField & DOCA
- EDA Licenses & Compute Farms
- Emulation & FPGA Prototyping Pipelines
- Firmware CI with Hardware in the Loop
- GPU Architecture & AI Hardware
- GPU Cluster Fabrics: InfiniBand & RoCE
Start free on Cariara · Coding problems · System design studio