vmstat & iostat

Overview

Overview vmstat and iostat provide system-wide and per-device performance statistics at a level of detail that top doesn't offer.

vmstat 1 (run every 1 second): The first row after the header is averages since boot — ignore it. Subsequent rows are 1-second samples.

Column groups and key fields: - procs: r = processes in run queue (waiting for CPU), b = processes in uninterruptible sleep (D state, blocked on I/O) - memory: swpd = swap used, free = free RAM, buff = buffer cache (block I/O), cache = page cache (file data). In Linux, free RAM is intentionally low — the OS uses it for cache - swap: si = swap in (KB/s, pages being read FROM swap TO RAM), so = swap out (KB/s, pages being WRITTEN TO swap). Any nonzero si/so means active swapping — serious performance problem - io: bi = blocks in (from disk), bo = blocks out (to disk), in 512-byte blocks/second - system: in = interrupts/sec, cs = context switches/sec. Very high cs (>100,000/sec) with low CPU can indicate excessive threading or system calls - cpu: identical to top's CPU columns — us sy id wa st

iostat -x 1 (extended I/O statistics per device): - r/s, w/s: read/write operations per second (IOPS) - rMB/s, wMB/s: throughput - rrqm/s, wrqm/s: merged requests (OS combining adjacent requests before sending to device) - r_await, w_await: average time (ms) for read/write I/O to complete including queue wait time - aqu-sz (or avgqu-sz): average queue length (requests waiting + being serviced) - %util: percentage of time the device is busy. Approaches 100% when the device is saturated

The critical insight: await is the total time from I/O request submission to completion. svctm (deprecated but historically present) was device service time. A large gap between await and svctm indicates queuing — the device can service I/O quickly but requests are piling up. With modern SSDs, %util can be 100% but the device is not truly saturated (SSDs queue internally).

Key concepts

vmstat r/b columns

r = run queue length (processes waiting for CPU — sustained >CPU_count means CPU bottleneck). b = blocked (uninterruptible D-state processes — sustained >0 means I/O bottleneck).

si/so for swap activity

si (swap in) and so (swap out) measured in KB/s. Any nonzero si/so means active swapping — RAM is full, OS is paging. Severe performance impact. Tune vm.swappiness to reduce.

iostat await vs %util

await: total I/O latency (ms) including queue wait. %util: device busy percentage. High await + high %util = disk saturated. High await + low %util = possible firmware or driver issue.

aqu-sz queue length

Average number of I/O requests in flight (queued + being serviced). Greater than 1 on HDDs indicates queuing. SSDs handle deeper queues efficiently (NVMe queue depth 64+).

When to use it

• Confirming whether swapping is occurring before it becomes critical
• Identifying disk I/O bottlenecks during database or file system issues
• Checking context switch rate for heavily threaded applications
• Baselining system behavior to detect anomalies

Interview questions

• You see vmstat showing si and so values consistently above 0. What does this mean and how urgent is it?
• Explain the difference between await and svctm in iostat -x output. What does a large gap between them indicate?

More Linux topics

• Text Processing Tools
• top & htop: Process Monitoring
• USE Method & Flame Graphs
• Users, Groups & Authentication
• Variables & Environment
• VFS & OverlayFS