ROCm & the AMD GPU Stack

A bottom-to-top tour of the AMD ROCm stack — firmware, amdgpu, amdkfd, ROCr, HIP, and math/comm libraries — and how firmware, driver, and ROCm versions must cohere on a validated node.

Overview

Why this topic

Every GPU stack covered so far on this platform has been NVIDIA plus CUDA. The AMD counterpart is ROCm (Radeon Open Compute), and it matters because AMD Instinct accelerators like the MI300X now run large training and inference fleets.

If you validate an AMD GPU CI/CD pipeline, your job is not one component. It is proving that GPU firmware, the amdgpu kernel driver and the ROCm userspace runtime all agree with each other on the same node.

The stack is a recipe

The stack layers cleanly from silicon up to the framework. Each layer only talks to the one directly below it, so a break at any layer looks like the whole GPU disappearing.

Framework — PyTorch / TensorFlow (ROCm build) — Your training or inference code
Libraries — rocBLAS, rocFFT, MIOpen, RCCL — Math and collective libraries
HIP runtime — libamdhip64 — CUDA-portable API
ROCr / HSA runtime — libhsa-runtime64 — Queues, signals, memory
Kernel drivers — amdkfd (/dev/kfd) and amdgpu (/dev/dri) — Compute and DRM
Firmware — Microcode — Loaded by amdgpu at probe
Silicon — AMD Instinct (CDNA, e.g. MI300) — The accelerator

The core validation problem

Three moving parts are released together and are only supported in specific combinations.

A node can look healthy at the OS level yet expose zero GPUs to HIP. Two common reasons:

Old firmware: the firmware the driver loaded is older than the runtime expects.
Stale driver build: dkms failed to rebuild amdgpu after a kernel bump.

hipErrorNoDevice — HIP calls and frameworks
Empty topology — /sys/class/kfd
GPU missing — rocminfo output

> [!IMPORTANT] Learning where each layer reports its version, and how a mismatch surfaces, is the entire discipline of this role.

Interview questions

Walk me through the ROCm stack layer by layer, and tell me exactly where a version mismatch breaks it.
You are handed a fresh MI300 node. How do you validate that firmware, driver, and ROCm are coherent before you let CI run on it?
Port this CUDA kernel to HIP. Show the mechanical steps and what actually changes.
CI reports no ROCm GPU detected on a node that worked yesterday. Debug it.
You own the compatibility matrix for a ROCm release. How do you reason about which firmware, driver, and ROCm versions are allowed together?
A multi-GPU training job on 8x MI300X hangs during AllReduce. How do you reason about RCCL and the interconnect?