Storage for GPU Training: Parallel Filesystems & GPUDirect Storage
Keeping GPUs fed and checkpoints fast: where training input stalls, parallel filesystems and local NVMe, GPUDirect Storage, checkpoint sizing from bytes per parameter, object storage tiers, dataloader tuning and measuring with fio.
Overview
Why this topic
A GPU that waits for data is the most expensive idle hardware in the building. Storage problems in training show up in two places: the input pipeline, where GPUs sit at low utilization because samples arrive too slowly, and checkpoints, where thousands of GPUs stop together while hundreds of gigabytes are written.
An interviewer wants to see that you can turn both into numbers: bytes per second the GPUs consume, bytes per checkpoint, the time budget for each, and the measurement that proves where the limit is.
Interview questions
- GPU utilization on a training job is 40 percent. How do you prove whether storage is the problem, and fix it?
- Size the checkpoint storage for a 70B model trained on 1,024 GPUs. How long may a checkpoint take, and what bandwidth does that need?
- What does GPUDirect Storage actually buy you, and how do you confirm a node is using it?
- How do you benchmark a storage system for a training cluster with fio, and what numbers do you compare?
- Design the data path for a 4,000-GPU cluster training on a 2 PB multimodal dataset kept in object storage.
- Checkpoints that took 90 seconds now take 12 minutes after the cluster doubled. Find the cause.
Sign up free to read the full topic
More AI Infrastructure topics
- Rack-scale NVLink: Operating GB200/GB300 NVL72
- ROCm & the AMD GPU Stack
- Serving LLMs On-prem: Triton, NIM, vLLM & the Model Store
- Silicon Root of Trust & Key Provisioning in Manufacturing
- Slurm for GPU Clusters
- Speculative Decoding & Parallel Token Generation
- Tape-out Release Engineering & Signoff Gates
- The Chip Design Flow as a CI Pipeline
- Verification Regressions at Scale
Start free on Cariara · Coding problems · System design studio