Linux Storage

Overview

Overview Linux presents storage to applications through a deep software stack that begins at the physical block device and surfaces to userspace as familiar POSIX file operations. Understanding the full path — block device, partition, filesystem, VFS, page cache, syscall — is critical for any engineer who needs to reason about performance, durability, or capacity on Linux systems.

The Virtual Filesystem Switch (VFS) is the kernel's unifying abstraction. It defines a common set of in-memory objects (superblock, inode, dentry, file) and function pointer tables (inode_operations, file_operations, address_space_operations) that every filesystem must implement. This design allows ext4, XFS, btrfs, tmpfs, and network filesystems to coexist and be mounted at different points in a single directory tree without the application caring which is which.

ext4 is the default filesystem on most Linux distributions. It extended the original ext2/3 design with extents (contiguous block ranges that replace indirect-block trees, dramatically reducing metadata for large files), a journal for crash consistency via the JBD2 layer, and delayed allocation (also called allocate-on-flush) to improve contiguity. Its three journal modes trade durability for performance: ordered mode protects against metadata corruption after a crash; writeback mode maximises throughput; full data journaling provides the strongest guarantee.

XFS was designed by SGI for high-concurrency, large-file workloads. It partitions the disk into allocation groups that each have their own inode allocator and free-space B-trees, allowing multiple threads to allocate simultaneously without a global lock. XFS pioneered delayed allocation and supports reflink (shared data extents, enabling instant CoW copies), sparse files, and direct I/O for database use cases. It does not support shrinking, and its lack of data journaling makes fsync-heavy workloads depend entirely on correct barrier ordering.

btrfs brings copy-on-write semantics to the filesystem layer. Every write goes to a new location, so the old version remains intact until a transaction commits. This enables atomic snapshots (a snapshot captures the B-tree root at a point in time in microseconds), subvolumes (independent filesystem trees within one btrfs pool), online defragmentation, and checksumming of both data and metadata. The CoW design also means that fragmentation grows over time on update-heavy workloads, and the recommended use case today is read-heavy or snapshot-heavy environments rather than high-IOPS databases.

LVM, md RAID, and overlay filesystems sit between raw block devices and the filesystem layer. LVM adds logical indirection: physical volumes are pooled into a volume group, and logical volumes are carved out on demand, with online resize, snapshots, thin provisioning, and live migration (pvmove) available. md provides software RAID, with RAID 1 for pure mirroring, RAID 5/6 for parity-based redundancy, and RAID 10 for the combined throughput and redundancy sweet spot. overlayfs layers two directory trees (lower read-only, upper read-write) into a unified mount, which is the mechanism Docker uses to implement image layers and container writable layers efficiently without copying.

Key concepts

ext4 Journal Modes and Inode Structure

ext4 is the most widely deployed Linux filesystem. Its journaling layer (JBD2) operates in three modes selected at mount time with the data= option. In ordered mode (the default), ext4 guarantees that data blocks reach their final on-disk location before the metadata transaction that references them is committed to the journal. This prevents the post-crash scenario where metadata says a block belongs to a file but the block contains garbage from a previous file. Ordered mode does not journal data, so a crash between the data write and the metadata commit can leave the file with its old size and old data, but it cannot expose another file's data. In writeback mode, ext4 only journals metadata. Data writes and metadata commits are independent. This is the fastest mode but means a crash can expose stale data blocks through newly committed metadata — acceptable for databases that manage their own consistency but dangerous for general-purpose use. In journal mode (data=journal), both data and metadata are written to the journal before being applied to the main filesystem. This gives the strongest durability guarantee but roughly halves write throughput because every write goes to disk twice. ext4 stores file data location using extents rather than the block-pointer tree of ext2/3. An extent is a (logical_block, physical_block, length) triple. Up to four extents fit directly in the inode, covering contiguous files up to about 512 MB with zero indirection. Larger files use an extent tree. This design dramatically reduces metadata for large sequential files compared to the triple-indirect block maps of ext3. A critical operational fact: ext4 reserves 5% of disk space for the root user by default. On a data volume this is wasted capacity. Set it to 1% or 0% with tune2fs -m.

XFS Allocation Groups and Delayed Allocation

XFS divides the filesystem into allocation groups (AGs). Each AG is a self-contained region of the disk with its own inode B-tree, free-space B-trees (one indexed by offset, one by size), and free inode B-tree. Because each AG is independent, multiple threads can allocate files simultaneously without a global lock, making XFS scale linearly with CPU count on multi-threaded create-heavy workloads. Delayed allocation (also called allocate-on-flush or delalloc) means XFS does not assign real disk blocks to a write until writeback time. When an application writes, XFS marks the range as "delayed" and tracks it in an in-memory extent map. At writeback the allocator sees the full contiguous region to be written and can find the best physical placement, improving spatial locality compared to allocating one block at a time. The trade-off is that file size reported by stat may be larger than allocated space until flush. XFS supports reflink, which lets two files share the same underlying data extents (like hard links but at the block level). A copy operation using cp --reflink=always completes in milliseconds regardless of file size. The first write to a shared extent triggers a copy-on-write split. This is used by btrfs-style snapshot workflows on XFS (since Linux 4.9). Direct I/O bypasses the page cache and issues DMA transfers directly to/from user buffers. XFS handles direct I/O particularly well because its allocation group parallelism means concurrent O_DIRECT writes from multiple threads do not serialize on a single allocator lock. PostgreSQL, MySQL InnoDB, and Cassandra all default to or recommend direct I/O on XFS. XFS cannot be shrunk online — plan the initial size carefully. It can be grown online with xfs_growfs after expanding the underlying block device.

btrfs Copy-on-Write, Subvolumes, and Snapshots

btrfs uses copy-on-write (CoW) for all writes. Rather than overwriting an existing block, btrfs writes the new data to a free location and updates the B-tree that maps logical addresses to physical addresses. The old block is freed only after the transaction commits. This design makes every write atomic at the filesystem level and enables instant snapshots. A snapshot in btrfs is just a new subvolume whose root B-tree node is a reference to the same root as the source subvolume at the moment of the snapshot. No data is copied. The two subvolumes share extents; a write to either triggers CoW, allocating new space only for the changed blocks. Subvolumes are independently mountable filesystem subtrees within one btrfs pool. Common layouts use a top-level subvolume named @ for the root filesystem and a subvolume @home for /home, allowing the root to be rolled back (by making @ point to a snapshot) without touching home data. The main operational hazard with btrfs is CoW fragmentation on database-style random-update workloads. Every small random write creates a new extent, scattering data across the disk. For databases, mount with nodatacow (disables checksumming and CoW for that file) or use a separate non-btrfs volume.

LVM: Physical Volumes, Volume Groups, Logical Volumes, and Thin Provisioning

LVM inserts a logical indirection layer between block devices and filesystems. A physical volume (PV) is a block device (or partition) initialized for LVM use. Multiple PVs are pooled into a volume group (VG), which maintains a mapping of 4 MB physical extents (PEs) across all member PVs. Logical volumes (LVs) are carved from the VG's PE pool and appear to the OS as regular block devices on which you format a filesystem. Thin provisioning allows LVs to be created larger than the actual available storage. A thin pool LV holds the actual data; thin LVs are virtual volumes that allocate real space on demand. This enables over-commitment: a VG with 100 GB can host five 50 GB thin LVs if average utilization is low. The risk is that if thin LVs collectively write more data than the pool holds, the pool runs out of space and thin LVs go into error state. pvmove migrates extents from one PV to another while the filesystem remains mounted and the system stays online. This is used to drain a disk before replacement without downtime. A key pitfall is forgetting to set up automatic thin pool extension. Add thin_pool_autoextend_threshold = 80 and thin_pool_autoextend_percent = 20 in /etc/lvm/lvm.conf and enable lvm2-monitor.service so the pool grows automatically before hitting 100%.

mdadm Software RAID

mdadm creates software RAID arrays from block devices managed by the Linux kernel's md (multiple devices) layer. The md layer sits below the filesystem and above device drivers, presenting a virtual block device to the filesystem. RAID is implemented entirely in kernel code, making it portable across hardware controllers. RAID 0 stripes data across N disks with no redundancy. Read and write throughput scale with N but a single disk failure destroys the array. RAID 1 mirrors data to N disks; reads can be parallelized across mirrors, but write throughput is limited by the slowest member. RAID 5 uses N-1 disks for data and one parity stripe distributed across all disks; it survives one disk failure. RAID 6 uses two independent parity schemes and survives two simultaneous disk failures at the cost of two disks' worth of overhead. RAID 10 (1+0) stripes across mirrored pairs, combining RAID 1 redundancy with RAID 0 throughput — the most common production choice for database servers. The write-intent bitmap (--bitmap=internal) records which stripes are being written. After a crash, only the stripes marked dirty need resyncing rather than the full array, dramatically reducing recovery time. Without it, a resync of a 4-disk RAID 10 array with 10 TB of data can take hours.

overlayfs, bind mounts, and tmpfs

overlayfs is a union filesystem that merges two directory trees: a read-only lower layer and a read-write upper layer. Reads are served from upper if the file exists there, otherwise from lower. Writes create new files or modified copies in upper (the copy-up operation). A workdir directory (on the same filesystem as upper) is used for atomic copy-up staging. The merged view is presented at the mount point. Docker uses overlayfs2 as its default storage driver. Each image layer maps to a lower directory. The running container adds an upper directory and a workdir. When the container writes to a file that exists in a lower layer, overlayfs copies the entire file to upper first (copy-up), then applies the write. This means the first write to a large file has latency proportional to the file size. Volumes (bind mounts or named volumes) bypass overlayfs entirely by mounting real host paths into the container, making them suitable for databases and any state that must survive container restarts. A bind mount re-exposes a directory (or file) from one location in the VFS tree to another without creating a new filesystem. It is the mechanism behind Kubernetes hostPath volumes and Docker -v /host/path:/container/path. tmpfs is an in-memory filesystem backed by anonymous memory and swap. It is used for /dev/shm (POSIX shared memory), /tmp on many modern systemd distros, and Kubernetes emptyDir volumes with medium: Memory. Because tmpfs data lives in RAM, it disappears on umount or reboot. overlayfs copy-up is a well-known performance gotcha for containers that modify large files (log rotation, SQLite databases, etc.). The solution is always to mount those paths as volumes, keeping hot write paths off the overlay stack.

/etc/fstab Options, UUIDs, and Mount Flags

The /etc/fstab file defines how block devices are mounted at boot. Each line specifies the device, mount point, filesystem type, options, dump flag, and fsck pass order. Using UUIDs instead of device names (/dev/sdb1) prevents breakage when disk enumeration order changes after adding hardware or rebooting after a kernel update. The UUID is stable across renames. Key mount options that affect performance and durability: noatime disables updating the access-time inode field on every read. On spinning disks this eliminates an extra write per read. For most workloads (databases, web servers) access time is irrelevant and noatime is a safe default. relatime (the kernel default since 2.6.30) is a compromise that updates atime only when it is older than mtime, avoiding most writes while remaining POSIX-compliant for applications that depend on atime ordering. barrier controls whether the filesystem issues write barriers (cache flush commands) to enforce ordering between journal commit and data writes. On drives with reliable write-back caches and power-loss protection, barrier=0 can improve throughput. On consumer drives without PLP, disabling barriers risks corruption after a power failure. nofail tells the kernel to continue booting even if the device is absent — essential for secondary data disks that should not prevent the system from starting. yaml Always run mount -a after editing fstab to catch syntax errors before the next reboot. A typo in fstab can drop a server into emergency mode.

When to use it

• Sizing and partitioning new servers where choosing between ext4, XFS, or btrfs depends on workload (sequential vs random, file sizes, snapshot requirements)
• Tuning database servers (PostgreSQL, MySQL, Cassandra) to use XFS with noatime, direct I/O, and appropriate readahead settings
• Designing LVM layouts for production databases where online resize, thin provisioning, and pvmove (zero-downtime disk migration) are required
• Building storage-resilient clusters using mdadm RAID arrays combined with LVM on top for flexible logical volume management
• Diagnosing disk-full errors where df and du disagree due to deleted-but-open file handles, reserved blocks, or sparse file accounting
• Architecting Docker or Kubernetes node storage, selecting between overlayfs, devicemapper, and btrfs storage drivers and understanding their durability trade-offs
• Implementing backup strategies using btrfs or LVM snapshots to provide crash-consistent point-in-time copies without stopping the application
• Investigating write latency spikes on SSDs where the culprit might be journaling mode, write-back cache policies, or kernel I/O scheduler selection

Interview questions

• Walk me through the full path of a write() system call from the application to the physical disk, including where data can be lost on a crash at each step.
• When would you choose XFS over ext4 for a database server, and what specific mount and filesystem options would you configure?
• What are the risks of LVM thin provisioning in production, and how do you mitigate them?
• How does overlayfs work, and how does Docker use it to implement image layers and container storage?
• Why might df and du report different values for the same directory, and how do you diagnose and resolve each cause?

More Linux topics

• Linux Filesystem
• Linux Kernel & /proc
• Linux Networking L2 and L3
• Linux Networking Tools
• Linux Processes
• Linux Signals
• lsof: List Open Files
• LUKS & dm-crypt Encryption
• LVM & Disk Management
• Memory Management
• Mount & fstab
• NUMA Topology & Memory Policy