perf & Flame Graphs

Overview

Overview perf is the Linux performance profiling Swiss Army knife, built directly into the kernel subsystem. It uses hardware performance counters and software events to profile at near-zero overhead.

perf stat: Collects hardware counter statistics for a command run. Key metrics: task-clock (milliseconds of CPU time), cycles (CPU clock cycles), instructions (retired instructions), IPC = instructions/cycles (efficiency measure — 1.0+ is good, below 0.5 suggests memory-bound workload), cache-misses (LLC misses indicate memory access pattern problems), branch-misses (mispredicted branches cause pipeline stalls).

perf record: Samples the call stack at a configurable rate (default 4000 Hz, often set to 99 Hz with -F 99 to avoid frequency aliasing). Records to perf.data. Flags: -g (call graph/stack traces), -p PID (attach to running process), -a (system-wide), --call-graph fp (frame pointer unwinding — fastest), --call-graph dwarf (DWARF unwinding — works without frame pointers but more overhead).

perf report: Interactive TUI for analyzing perf.data. Shows functions sorted by CPU time with caller/callee breakdown.

Flame Graphs (invented by Brendan Gregg): A visualization where the x-axis represents alphabetically sorted stack frames (width = proportion of time), y-axis is call stack depth (bottom = root, top = leaf function). The key insight: wide frames at the top are where time is actually spent. Color is irrelevant. Look for wide plateaus that indicate hot code paths.

Generating flame graphs: 1. perf record -F 99 -g -p PID -- sleep 30 2. perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg

Off-CPU analysis: Regular CPU profiling misses time spent waiting (I/O, locks, sleep). Off-CPU flame graphs show where threads block, not just where they compute.

Key concepts

perf stat hardware counters

perf stat cmd — shows cycles, instructions, IPC, cache-misses. IPC < 0.5 suggests memory-bound. High cache-misses suggest poor data locality. Branch-misses suggest unpredictable conditions.

IPC (instructions per cycle)

IPC = instructions / cycles. > 1.0: CPU well utilized, efficient code. 0.5-1.0: moderate efficiency. < 0.5: memory-bound or poor branch prediction. Modern CPUs can achieve IPC 3-4 with ideal workloads.

Flame graph width=time

Wider frames = more CPU time. Look for wide plateaus at the top of the flame (leaf functions). Alphabetical x-axis — position is irrelevant. Merge identical stacks for accurate width.

Off-CPU analysis

CPU profiling shows where CPU runs. Off-CPU profiling shows where threads wait (I/O, locks, sleep). Use perf trace or bpftrace offcputime.bt for blocking time analysis.

When to use it

• Identifying CPU hotspots in production services
• Investigating high CPU usage with no obvious cause
• Comparing performance before and after optimization (differential flame graphs)
• Finding lock contention in multithreaded applications

Interview questions

• How do you generate a CPU flame graph on Linux? Walk through the entire process from profiling to visualization.
• A service has high CPU usage but flame graphs show no obvious hotspot — it's spread thin. What techniques would you use next?

More Linux topics

• LVM & Disk Management
• Memory Management
• Mount & fstab
• NUMA Topology & Memory Policy
• Package Management
• Packet Capture with tcpdump
• Pipes & Redirection
• seccomp Syscall Filtering
• SELinux
• Service Management with systemctl
• Socket Statistics: ss & netstat
• Software RAID with mdadm