RED, USE & Four Golden Signals

Three named methodologies for what to measure. Pick by what you're monitoring.

Overview

Three methods for "what to measure"

Three named methods answer "what should I measure?", and engineers conflate them constantly. Each targets a different kind of thing.

RED Method, for services

Tom Wilkie (formerly Weaveworks, now Grafana). If you can answer these three for any service, you can tell whether it's healthy. It maps cleanly onto Prometheus histogram metrics.

Rate — Requests per second
Errors — Count or % of failed requests
Duration — Distribution of response latencies (p50, p95, p99)

USE Method, for resources

Brendan Gregg (Netflix, then Intel). For CPU, memory, disk, network and queues.

Utilization — % busy or % full
Saturation — Queue depth or wait time: work backlog beyond utilization
Errors — Error counts

> [!TIP] CPU utilization tells you the average; CPU saturation (load average, runnable threads) tells you the queue beyond it. 80% utilization with 0 saturation is fine; 80% with deep saturation is overloaded.

Four Golden Signals, Google's superset

Google SRE Book Ch 6: "If you can only measure four metrics of your user-facing system, focus on these four."

Latency — Time to serve a request, with success and error latency separated
Traffic — Demand on the system (req/sec, MB/sec)
Errors — Explicit failures, plus implicit ones such as slow successes that miss the SLO
Saturation — How full the service is (queue depth, memory headroom)

How to choose

User-facing service — Four Golden Signals, or RED plus saturation
Backend resource (DB host, message broker) — USE
Microservice mesh — RED everywhere; USE on each underlying host
Anything else — Four Golden Signals; they generalize

> [!IMPORTANT] These methods are about completeness, not exact metrics. If you have rate but not errors, you'll learn about an outage from the customer.

Key concepts

RED Method: For services. Rate, Errors, Duration. Tom Wilkie. Maps to Prometheus histograms.
USE Method: For resources. Utilization, Saturation, Errors. Brendan Gregg. Resource-level observability.
Four Golden Signals: Google SRE Book. Latency, Traffic, Errors, Saturation. The superset; generalizes to most systems.
Saturation: Queue depth or wait time beyond utilization. Leading indicator: 80% CPU might be fine, 80% CPU + load average 30 means overload.
Latency: success vs error separated: Don't average them. Errors often fail fast; mixing them with successes makes the latency distribution lie.

When to use it

Designing dashboards or alerts for a new service — pick the appropriate method
Reviewing existing observability — checking that all four (or three) signals are actually instrumented
Capacity planning — saturation is the leading indicator
Interview answers about "what would you monitor for a service like X"

Interview questions

Walk me through RED, USE, and the Four Golden Signals — when do you use each?
Why is saturation so often missed?
Why must success and error latency be tracked separately?