Kubernetes Storage

Overview

Overview

Kubernetes was designed for stateless workloads, but production systems need durable storage for databases, message queues, file stores and logging pipelines. The storage subsystem has matured through three generations.

PV and PVC

• PersistentVolume (PV) — Cluster — Actual storage capacity
• PersistentVolumeClaim (PVC) — Namespace — A request for storage by capability, not by name or location
• StorageClass — Cluster — Enables dynamic, self-service provisioning; no pre-created PVs

Access modes

• ReadWriteOnce (RWO) — Read-write from a single node — Cloud block devices: AWS EBS, GCP Persistent Disk
• ReadOnlyMany (ROX) — Read-only from many nodes — Static content
• ReadWriteMany (RWX) — Read-write from many nodes — Networked or distributed: NFS, CephFS, Azure Files
• ReadWriteOncePod (RWOP) — Read-write from a single pod — Enforced at the Kubernetes API layer, not only by the backend

StorageClass settings

• reclaimPolicy — Retain — Production databases; prevents accidental data loss, needs manual cleanup
• reclaimPolicy — Delete — Ephemeral or easily recreated data
• volumeBindingMode — Immediate — Provisions on PVC creation; may pick a zone with no schedulable nodes for the workload
• volumeBindingMode — WaitForFirstConsumer — Waits until a pod is scheduled so topology is known; recommended for most cloud block storage

CSI drivers

The CSI model splits storage logic in two. Controller components handle cluster-wide work such as provisioning and snapshots; node components mount and unmount on each node. Standard sidecars from Kubernetes SIG Storage translate API events into CSI gRPC calls, so vendors only implement the gRPC interface in one plugin binary.

> [!TIP] CSI migration is the ongoing project to replace every in-tree volume plugin with an equivalent CSI driver while keeping the user-facing API identical.

Snapshots and StatefulSets

• VolumeSnapshot (snapshot.storage.k8s.io) — Point-in-time backup and restore on any CSI driver that implements the snapshot calls; with Velero, application-consistent backups
• StatefulSet volumeClaimTemplates — A dedicated PVC per replica with predictable names; the foundation of most database deployments on Kubernetes

Key concepts

PersistentVolume Lifecycle

A PersistentVolume goes through five phases: Available (provisioned but not yet bound), Bound (matched to a PVC), Released (the PVC was deleted but the PV has not been reclaimed), Failed (automatic reclamation failed), and implicitly Pending when dynamic provisioning is in flight. The transition from Released to the next state depends on the reclaimPolicy. With Retain, the PV stays in Released indefinitely until an administrator manually deletes the PV object and the backing storage. With Delete, the external-provisioner sidecar automatically calls DeleteVolume on the CSI driver and then deletes the PV object. Static provisioning means an administrator pre-creates the PV with a specific capacity and access mode before any PVC exists. The binding loop matches PVCs to PVs by finding a PV whose capacity is at least as large as the PVC request and whose access modes are a superset of what the PVC needs. Dynamic provisioning creates the PV on demand in response to a PVC that names a StorageClass; no pre-created PV is needed. Understanding each phase is critical for debugging stuck PVCs and for designing disaster recovery procedures that do not accidentally delete production data.

StorageClass Configuration

A StorageClass is the factory specification for PVs created dynamically. The provisioner field names the CSI driver that will handle CreateVolume calls. The parameters map passes driver-specific options such as disk type, IOPS tier, or encryption key. reclaimPolicy and volumeBindingMode are the two most operationally significant fields. Setting volumeBindingMode to WaitForFirstConsumer is essential when using zonal block storage in a multi-zone cluster. Without it, a PVC for a PostgreSQL pod might be provisioned in us-east-1a while the pod gets scheduled to us-east-1b, causing the pod to stay in Pending forever because the volume cannot be attached across zones. allowVolumeExpansion: true enables users to edit a PVC's storage request upward after creation. The CSI driver must implement the NodeExpandVolume and ControllerExpandVolume RPCs for online expansion to work. Most major cloud CSI drivers support this for ext4 and xfs filesystems.

Access Modes and Volume Modes

Access modes express the concurrency contract between a volume and its mounters. ReadWriteOnce (RWO) permits read-write by one node; it is the mode supported by nearly all block storage backends and is the right choice for a single-instance database. ReadOnlyMany (ROX) allows many nodes to mount read-only; useful for content that is written once and read by many pods. ReadWriteMany (RWX) is the most demanding mode and requires a networked filesystem or distributed storage backend; NFS, CephFS, Azure Files, and AWS EFS via the efs-csi-driver all support RWX. ReadWriteOncePod (RWOP), introduced in Kubernetes 1.22 and graduated to stable in 1.29, restricts read-write access to exactly one pod even if multiple pods on the same node request it. Volume mode controls whether the volume is presented as a formatted filesystem (Filesystem, the default) or as a raw block device (Block). Raw block mode is used by storage systems like Ceph RBD that do their own internal formatting, or by databases like PostgreSQL when configured to bypass the OS filesystem cache. A pod that uses a Block-mode PVC references the volume with a volumeDevices entry instead of volumeMounts, and the device path appears under /dev inside the container rather than a directory path.

CSI Sidecar Architecture

The CSI driver for a storage backend is split into two plugin types: a controller plugin that runs as a Deployment and handles cluster-wide operations, and a node plugin that runs as a DaemonSet on every node and handles local mount operations. The Kubernetes SIG Storage team maintains a set of external sidecar containers that translate Kubernetes API events into CSI gRPC calls, so the storage vendor only ships the gRPC server binary. The external-provisioner sidecar watches for PVCs that request a matching StorageClass and calls CreateVolume on the CSI controller plugin. The external-attacher watches VolumeAttachment objects and calls ControllerPublishVolume (attach) and ControllerUnpublishVolume (detach). The external-resizer watches PVC edits and calls ControllerExpandVolume followed by NodeExpandVolume via the node plugin. The external-snapshotter manages VolumeSnapshot objects and calls CreateSnapshot and DeleteSnapshot. On the node side, the node-driver-registrar registers the CSI node plugin socket with kubelet via the kubelet plugin registration mechanism so that kubelet knows which socket to use for NodeStageVolume and NodePublishVolume calls. The liveness-probe sidecar calls the CSI Probe RPC on a timer and reports unhealthy if the plugin does not respond, enabling automatic pod restart.

StatefulSet volumeClaimTemplates

StatefulSets use volumeClaimTemplates to automatically provision a dedicated PVC for each pod replica. The template is a PVC spec embedded in the StatefulSet spec. When the StatefulSet controller creates a pod with ordinal index N, it also creates a PVC named --N in the same namespace. These PVCs persist across pod deletions and rescheduling, giving each replica stable storage identity that matches its stable network identity. Critically, PVCs created by a StatefulSet are not deleted when the StatefulSet is scaled down or deleted. This is a deliberate safety decision to prevent data loss. Administrators must manually delete the PVCs when decommissioning a StatefulSet permanently. This creates PVCs named data-postgres-0, data-postgres-1, and data-postgres-2, each bound to a separate PV in a topology-aware zone.

VolumeSnapshot and Restore

VolumeSnapshot support is provided through three CRDs: VolumeSnapshotClass (driver and deletion policy), VolumeSnapshotContent (the actual snapshot in the backend), and VolumeSnapshot (the user-facing request). The workflow mirrors the PV/PVC pattern. Creating a VolumeSnapshot triggers the snapshot controller to call CreateSnapshot on the CSI driver via the external-snapshotter sidecar. A VolumeSnapshotContent is created and bound to the VolumeSnapshot. Restore works by creating a new PVC with a dataSource that references the VolumeSnapshot. The CSI provisioner detects the dataSource and calls CreateVolume with a VolumeContentSource pointing to the snapshot, causing the backend to clone snapshot data into a new volume. The new PVC binds normally and can be mounted by a pod. The deletion policy on VolumeSnapshotClass controls whether deleting a VolumeSnapshot also deletes the VolumeSnapshotContent and the backend snapshot (Delete) or retains them (Retain).

CSI Migration

CSI migration is the process of replacing Kubernetes in-tree volume plugins (code inside the kube-controller-manager and kubelet binaries) with external CSI drivers while keeping the existing PV, PVC, and StorageClass manifests unchanged. When CSI migration is enabled for a plugin, the in-tree code becomes a shim that translates in-tree API calls to the equivalent CSI gRPC calls directed at the installed CSI driver. Migration is controlled per-plugin via feature gates. As of Kubernetes 1.29, migrations for AWS EBS, GCE PD, Azure Disk, Azure File, Cinder, and vSphere are all enabled by default (GA). Migration can be disabled per-node with the CSIMigrationDisable feature gate if a node does not have the CSI driver installed, giving a gradual rollout path. The practical implication is that operators must install the appropriate CSI driver before enabling migration, and must ensure the driver version supports all features (snapshots, resize) that the workloads depend on.

When to use it

• Running a relational database such as PostgreSQL or MySQL inside Kubernetes where data must survive pod restarts and rescheduling
• Deploying a distributed stateful system such as Kafka, Elasticsearch, or Cassandra using a StatefulSet and per-pod PVCs
• Implementing a backup and restore workflow using VolumeSnapshots to create point-in-time copies of database volumes
• Using WaitForFirstConsumer to ensure cloud block volumes are provisioned in the same availability zone as the pod that will use them
• Migrating from an in-tree volume plugin to a CSI driver without changing any application manifests
• Sharing read-only reference data across many pods in different nodes using a ReadOnlyMany volume backed by NFS or a CSI driver that supports ROX
• Enforcing exclusive single-pod access to a volume beyond what RWO provides by using ReadWriteOncePod
• Expanding a volume online without downtime by enabling allowVolumeExpansion in a StorageClass and editing the PVC resource request

Interview questions

• A PVC has been in Pending state for 10 minutes. Walk me through how you would diagnose and resolve it.
• Explain the role of each CSI sidecar container and why they are separate processes rather than part of the main driver binary.
• When should you use WaitForFirstConsumer instead of Immediate for volumeBindingMode, and what are the tradeoffs?
• Describe the full VolumeSnapshot restore workflow from snapshot creation to mounting the restored volume in a pod.
• What is the difference between ReadWriteOnce and ReadWriteMany, and which storage backends support each? What happens if you use the wrong one?

More DevOps topics

• Kubernetes Pod Scheduling
• Kubernetes RBAC
• Kubernetes Runtime Classes
• Kubernetes Secrets Management
• Kubernetes Security: Pod Security and Runtime Hardening
• Kubernetes Services
• Kubernetes the Hard Way
• Kubescape and Kubernetes Runtime Security
• Kustomization and Reconciliation
• Lease and Reservation Control Planes
• libstdc++, libc++, glibc and Symbol Versioning
• Lifecycle Hooks and Tests