Slurm for GPU Clusters

Overview

Why Slurm

Most on-prem GPU training clusters, and most supercomputers, schedule with Slurm. It treats a node as a set of CPUs, memory and generic resources (GRES) such as GPUs, keeps a queue per partition, and starts each job only when every node it needs is free at once. For multi-node training that all-or-nothing start (gang scheduling) is exactly what you want.

• slurmctld — Controller (plus a backup) — Queue, priorities, placement, node states
• slurmd — Every compute node — Starts job steps, reports health and resources
• slurmdbd — Accounting host — Usage history for fair share, sacct, QOS limits
• slurmrestd — Optional — REST API for portals and automation

Interview questions

• Configure a 32-node H100 cluster in Slurm so jobs get exactly the GPUs they ask for, on well-placed nodes.
• Launch a 4-node, 32-GPU PyTorch job with srun and torchrun. What does the batch script look like, and what goes wrong?
• A node shows drain with the reason "gres/gpu count reported lower than configured (7 < 8)". Walk me through it.
• Explain how priority, fair share, QOS and preemption interact. A research team says their jobs never start; how do you find out why?
• Write a health check for a GPU node. What does it test, and how does it interact with Slurm?
• Jobs on one node keep failing with "Kill task failed" and the node drains. What is happening?

More AI Infrastructure topics

• PyTorch torch.compile, FSDP2 & Training Optimization
• Quantization & Model Compression
• Rack-scale NVLink: Operating GB200/GB300 NVL72
• ROCm & the AMD GPU Stack
• Serving LLMs On-prem: Triton, NIM, vLLM & the Model Store
• Silicon Root of Trust & Key Provisioning in Manufacturing
• Speculative Decoding & Parallel Token Generation
• Storage for GPU Training: Parallel Filesystems & GPUDirect Storage
• Tape-out Release Engineering & Signoff Gates
• The Chip Design Flow as a CI Pipeline
• Verification Regressions at Scale