GPU Fleet Health: XID Errors, ECC & DCGM
Reading GPU faults in a running fleet: XID codes, ECC and row remapping, throttle reasons and NVLink errors, DCGM metrics and alerts, and the pipeline that cordons, resets or returns a bad GPU.
Overview
Why this topic
A GPU fleet fails every day. With thousands of GPUs, some GPU somewhere is logging an XID, accumulating ECC errors or throttling right now. The job of fleet health is to tell the faults that need a part replaced from the ones that are an application bug, act on each one automatically, and never leave a bad GPU in a training job.
The raw signals come from three places: the kernel log (XID lines from the NVIDIA driver), NVML (what nvidia-smi reads), and DCGM (NVIDIA's Data Center GPU Manager, which watches, diagnoses and exports metrics).
Interview questions
- Design an automated remediation pipeline for GPU faults in a Kubernetes or Slurm fleet.
- A node logs XID 79. Walk me through what you do.
- Explain ECC errors, row remapping and when a GPU should be replaced.
- Write the Prometheus alert rules you would put on dcgm-exporter metrics.
- One GPU in a job runs 20 percent slower than the others, with no errors. How do you find out why?
- How do you tell an application bug from a hardware fault when you see XID 13 or 31?
Sign up free to read the full topic
More AI Infrastructure topics
- EDA Licenses & Compute Farms
- Emulation & FPGA Prototyping Pipelines
- Firmware CI with Hardware in the Loop
- GPU Architecture & AI Hardware
- GPU Cluster Fabrics: InfiniBand & RoCE
- GPU Driver & Software Stack Release Validation
- GPU Node Bring-up & Burn-in
- GPU Quotas, Utilization & Chargeback On-prem
- GPU Virtualization: Passthrough, vGPU, SR-IOV & MIG in VMs
- GPUs on Kubernetes: GPU Operator, Device Plugin, MIG & Time-slicing
- LLM Inference: KV Cache, Batching & Scheduling
- Manufacturing Test Data & Yield Pipelines
Start free on Cariara · Coding problems · System design studio