USE Method & Flame Graphs

Brendan Gregg USE Method for resource analysis, flame graph interpretation, off-CPU analysis, and production perf workflow.

Overview

Brendan Gregg's USE Method provides a systematic checklist for resource performance analysis. For every resource, measure:

U — Utilization — time resource was busy (0-100%). High utilization doesn't always mean saturation.
S — Saturation — work queued that cannot be serviced (run queue depth, I/O queue depth). Saturation causes latency.
E — Errors — hardware or software errors (disk errors, dropped packets, CPU machine checks).

Resources to check: CPU, memory, network interfaces, storage devices, controllers, interconnects.

Flame graphs visualize stack traces over time, showing where CPUs spend their time:

x-axis — alphabetical (not time). Width = proportion of CPU samples in that function.
y-axis — stack depth. Bottom = on-CPU, top = leaf function.
Wide flat tops — the function where CPU is spending time (the bottleneck).
Wide bases — a common code path.

On-CPU flame graphs miss time spent blocked (waiting for I/O, locks, sleep). Off-CPU analysis captures time processes spend not executing:

Utilization vs Saturation: A disk at 70% utilization has headroom. The same disk with a queue depth of 32 (saturation) causes queuing latency even at 70% utilization. Both metrics are needed.
Flame graph width: Width of a frame represents the proportion of CPU samples where that function was on the call stack. The widest frames at the top of the y-axis are bottlenecks.
Off-CPU time: Time a thread is descheduled — waiting for I/O, sleeping, or blocked on a lock. Not visible in CPU flame graphs. Requires scheduler event tracing (sched_switch tracepoint).
iowait (%wa): Time CPUs are idle but at least one process is blocked waiting for I/O. High iowait means storage is the bottleneck, not the CPU.

Systematically triaging a performance problem on an unfamiliar system
Identifying which function is consuming the most CPU time in a service
Explaining why a service has high latency despite low CPU utilization (off-CPU / I/O wait)

Walk through how you would diagnose a production service with high latency but low CPU usage.