Disk IOPS Saturation

Overview

Overview

Disk IOPS saturation happens when storage cannot serve reads and writes as fast as the application issues them, so pending I/O queues up. You see high I/O wait (the wa column in top), slow database queries and disk latency that surfaces as application timeouts. Unlike CPU saturation, which users mostly feel as latency, I/O saturation can cascade as write buffers fill and threads block on I/O.

Reading iostat -x 1

• await — Average ms an I/O waits before it is serviced; high values mean saturation
• svctm — Average service time per I/O; deprecated but often still shown
• %util — Share of time the device is busy; consistently above 80-90% means saturation
• r/s, w/s — Read and write operations per second

EBS volume types

• gp2 — 3 IOPS per GB baseline (100GB gets 300), bursts to 3,000 while credits last — Credit-based; throttles to baseline when credits run out
• gp3 — Fixed 3,000 baseline regardless of size; provision up to 16,000 at extra cost — Throughput configurable up to 1,000 MB/s
• io2 — Up to 64,000 IOPS with consistent sub-millisecond latency — For I/O-heavy databases such as Oracle, SQL Server, high-transaction PostgreSQL

Match storage to the I/O pattern

• OLTP: web app databases, API backends — Random: small reads and writes scattered across the volume — IOPS, with low latency per operation
• Analytics and reporting — Sequential: large contiguous reads — Throughput (MB/s); IOPS count matters less

Reduce the IOPS you need

• Connection pooling — Fewer connections, fewer concurrent I/O paths
• synchronous_commit = off for non-critical writes — Fewer synchronous WAL flushes
• wal_compression = on — Fewer WAL bytes written
• checkpoint_completion_target = 0.9 — Checkpoint writes spread out instead of spiking
• Read replicas — Read I/O spread over more volumes
• Indexes that remove full table scans — Multi-page random I/O becomes single-page lookups

> [!TIP] Enable RDS storage autoscaling with a maximum 3-4x the current allocation, and alarm on free storage at 20% so autoscaling finishes before the disk fills.

Key concepts

IOPS

Input/Output Operations Per Second — the number of read or write operations a storage device can service per second; the primary capacity metric for random I/O workloads.

await

The average time in milliseconds that I/O requests spend waiting in the device queue plus the service time — the key saturation indicator in iostat output.

BurstBalance

CloudWatch metric for gp2 EBS volumes tracking remaining IOPS burst credits as a percentage — when it reaches 0, the volume is throttled to the baseline rate.

gp3

The current-generation AWS general-purpose SSD EBS volume type with a fixed 3,000 IOPS baseline (not credit-based like gp2), configurable to 16,000 IOPS independently of volume size.

I/O Wait

The percentage of time CPU cores are idle waiting for pending I/O operations to complete — shown as the wa column in top; consistently above 20-30% indicates I/O saturation.

When to use it

• Diagnosing slow database queries that correlate with high disk await times
• Explaining the difference between gp2 burst IOPS and gp3 provisioned IOPS
• Investigating why RDS instance CPU is low but queries are slow (storage bottleneck)
• Choosing the right EBS volume type for a specific database workload profile
• Designing a storage architecture that distributes I/O across read replicas

Interview questions

• How do you diagnose disk I/O saturation on a Linux server and identify the bottleneck?
• When should you use gp3 vs io2 EBS volumes for a database workload?
• How does PostgreSQL WAL configuration affect disk IOPS and what settings reduce write I/O?

More Troubleshooting topics

• CoreDNS Failures in Kubernetes
• CPU Runaway Process Investigation
• CrashLoopBackOff
• Database Deadlock Diagnosis
• Database Migrations in Production
• Deployment Rollback Strategies
• DNS Failures
• EC2 Instance Unresponsive
• ECS Task Failures and Restarts
• ELB 5xx Errors
• Flaky CI/CD Pipeline Failures
• HPA Not Scaling