Storage Errors — FailedMount, Pending PVCs, Multi-Attach and Full Volumes

Volumes that will not provision, attach, mount, fit or grow: Pending claims, attach and mount failures, Multi-Attach, full disks and stuck resizes, with the commands to prove each one and how it differs on EKS, AKS and GKE.

Overview

Storage is the most location-bound part of Kubernetes. A block disk lives in one availability zone, most block disks attach to one node at a time, and every node can hold only a limited number of them. Most storage incidents come down to one of those three facts, plus a fourth: Kubernetes does not watch how full a volume is, so a full disk looks perfectly healthy to the cluster.

A volume goes through a fixed path before a container can write to it: the claim is provisioned (a disk is created and a PersistentVolume bound), the pod is scheduled to a node that can use that disk, the disk is attached to the node, the kubelet mounts it into the pod, and later the disk may be expanded. Each stage is run by a different component and reports its failure on a different object: provisioning on the PVC, scheduling on the pod as FailedScheduling, attach on the pod as FailedAttachVolume and in the VolumeAttachment, mount on the pod as FailedMount. Read the event, find the stage, and you know which component to look at.

The one idea to keep: find the stage first, then the component. kubectl describe pvc and kubectl describe pod tell you which stage stopped; the CSI driver's logs and the cloud's own disk API tell you why.

Symptom to first command

Pod stuck ContainerCreating, FailedAttachVolume or FailedMount — Disk still attached elsewhere, attach limit, CSI node plugin down, cloud permissions — kubectl describe pod -n prod
PVC stays Pending — Missing StorageClass or default, provisioner failing, or WaitForFirstConsumer waiting for a pod — kubectl describe pvc data-orders -n prod
Multi-Attach error or Waiting for detach on the new pod — ReadWriteOnce disk still attached to the old pod's node — kubectl describe pvc data-orders -n prod (the Used By line)
App fails with No space left on device, pod still Ready — Volume full of bytes or of inodes — kubectl exec -n prod -- df -h /data and df -i /data
PVC resized but the pod still sees the old size — Node-side filesystem expansion pending or failed — kubectl get pvc data-orders -n prod -o jsonpath='{.status.conditions}'
MountVolume.SetUp slow, or permission denied writing the volume — fsGroup recursive ownership change, missing fsGroup, or SELinux labels — kubectl get pod -n prod -o jsonpath='{.spec.securityContext}'
driver name ... not found in the list of registered CSI drivers — CSI node plugin not running or not registered on that node — kubectl get csinode -o yaml
Pod Pending with didn't match PersistentVolume's node affinity — Disk is in a zone with no schedulable node — kubectl get pv -o jsonpath='{.spec.nodeAffinity}'

What you will be able to do

Name the stage a storage failure belongs to from the event text alone, and go straight to the right component.
Unstick a volume held by a dead node the supported way (the out-of-service taint), and explain the 6-minute force detach.
Prevent zone mismatches, Multi-Attach and surprise full disks with StorageClass settings, rollout strategy, policy and alerts.
Walk a volume expansion through its controller and node stages and recover when it fails.
Say what changes on EKS, AKS and GKE: which driver, which limits, which parts the provider runs for you.

Key concepts

PersistentVolumeClaim (PVC): A request for storage by size, access mode and class. It binds to one PersistentVolume, which represents the real disk or share.
StorageClass: The template for dynamic provisioning: which CSI driver, which disk parameters, the binding mode and whether expansion is allowed.
WaitForFirstConsumer: A binding mode that delays creating the disk until a pod using the claim is scheduled, so the disk is made in that pod's zone.
Access modes: ReadWriteOnce allows one node, ReadWriteOncePod allows one pod (stable since 1.29), ReadWriteMany allows many nodes and needs file storage.
VolumeAttachment: The object that records that a CSI volume should be, or is, attached to a node. It is the Kubernetes side of the cloud attach call.
CSI controller and node plugins: The controller side (provisioner, attacher, resizer sidecars) talks to the cloud; the node side, a DaemonSet, formats, mounts and expands on each node.
CSINode: Per-node record of registered CSI drivers and how many volumes each can attach; the scheduler uses it for attach limits.
Out-of-service taint: node.kubernetes.io/out-of-service added by an operator to a shut-down node, so its pods are force-deleted and their volumes detached at once.
Two-stage expansion: The controller grows the disk and the PV; the node grows the filesystem and only then updates the PVC capacity. A stall between them shows as FileSystemResizePending.
VolumeAttributesClass: A class of mutable disk settings such as IOPS and throughput, stable since 1.34, applied to an existing claim through spec.volumeAttributesClassName.

When to use it

A pod is stuck in ContainerCreating and its events mention FailedAttachVolume, FailedMount or Multi-Attach.
A PersistentVolumeClaim stays Pending and you need to know whether that is expected or broken.
An application reports No space left on device while its pod looks healthy.
A volume expansion finished in the cloud but the pod still sees the old size.
A node died and its stateful pods cannot move to another node.
You are preparing for an SRE or platform interview that covers stateful workloads on EKS, AKS or GKE.

Interview questions

A node running orders-0 of a StatefulSet lost power. The pod shows Terminating and no replacement starts. Walk me through recovering it.
Our Postgres pod is Running and Ready, but the app logs No space left on device. How do you handle it, and what would have caught it earlier?
We raised a PVC from 20Gi to 50Gi an hour ago. kubectl get pv says 50Gi but the database still sees 20G. What is going on?
After a deploy, the new pod of our single-replica orders Deployment is stuck in ContainerCreating with a Multi-Attach error, and the old pod is still running. Why, and how do you fix it for good?
How does debugging storage differ on EKS, AKS and GKE?
The cluster autoscaler removed nodes in one zone overnight. This morning a StatefulSet pod is Pending with a node affinity message, and the autoscaler is not adding nodes. Walk me through it.
A new analytics pod with a volume of several million small files takes many minutes to start and sits in ContainerCreating. Nothing is failing. What is happening?