SLAs, SLOs, and Percentile Latency

How to state, measure and defend latency targets with percentiles instead of averages, why fan-out makes the slow tail worse, and the techniques that shorten it.

Overview

Users do not feel your average response time. They feel their own request, and the slow ones are what they remember. So a serious latency target is written as a percentile: '99% of requests finish within 200 ms', not 'the average is 30 ms'.

This topic covers the vocabulary (SLI, SLO, SLA), why averages mislead, why the slow tail gets worse as one request fans out to many servers, how to measure percentiles correctly, and the main ways to shorten the tail.

SLI, SLO and SLA

Three terms from Google's SRE practice are now standard. An SLI (service level indicator) is a measurement. An SLO (service level objective) is a target for that measurement. An SLA (service level agreement) is a promise to users that comes with consequences if the SLO is missed, such as service credits.

SLI — A measured number — p99 latency of checkout requests over 5 minutes
SLO — A target for the SLI — p99 under 300 ms, measured over 28 days
SLA — A contract with consequences — An illustrative contract: below 99.9% success in a month earns a service credit
Error budget — How much the SLO lets you miss — A 99.9% SLO allows 0.1% of requests to fail or be slow

Set the internal SLO tighter than the SLA. The gap gives the team time to fix a problem before it costs money. The error budget turns reliability into a shared decision: while budget is left, teams can ship quickly; when it is spent, releases slow down and the work shifts to stability.

Why the average hides the problem

A percentile is read from the sorted list of response times. The p50 (median) is the time half of the requests beat. The p99 is the time 99% beat, so 1 request in 100 is slower. The average mixes fast and slow requests into a number no user actually saw.

Mean — 29.1 ms — Looks healthy, yet no request took about 29 ms
p50 — 10 ms — The typical request is fast
p90 — 10 ms — Most requests are fast
p99 — 1,500 ms — More than 1 in 100 requests takes a second and a half

The slow requests often belong to the most valuable users. Amazon's Dynamo paper explains that customers with long histories need more processing, so they land in the tail. That is why Amazon's services state their latency SLAs at the 99.9th percentile; the paper says going higher was judged too costly for the gain.

Response time, service time and queueing

Response time is what the client measures: network time, time waiting in a queue, and service time (the time the server actually works on the request). Many people use latency to mean response time; some authors reserve latency for the waiting part only. Say which one you mean.

Queueing explains much of the tail. If a few slow requests hold all of a server's worker threads, fast requests behind them wait. This is head-of-line blocking: the fast requests look slow though their own work is quick. Measure response time on the client side, or you will not see this wait.

Fan-out makes the tail everyone's problem

When one user request calls many servers in parallel and waits for all of them, it is as slow as the slowest one. If each server is slow for 1 request in 100, the chance that at least one of n calls is slow is 1 − 0.99^n.

1 — 1%
10 — 9.6%
20 — 18%
50 — 39%
100 — 63%

Google's 'The Tail at Scale' paper uses exactly this case: servers that usually answer in 10 ms but take one second at the 99th percentile. With 100 of them per request, 63% of user requests take over a second. In a real Google service it measured, the p99 for one leaf request was 10 ms, but the p99 for all leaf requests to finish was 140 ms.

Measuring percentiles correctly

You cannot keep every response time forever, and you cannot combine percentiles by averaging them. The usual tool is a histogram: counts of requests in latency buckets, which can be added across servers and time windows, and from which percentiles are then estimated.

HdrHistogram — Buckets sized to keep a chosen number of significant digits across the whole range — Constant memory and constant-time recording; can correct for coordinated omission
t-digest — A small sketch that keeps more detail near the tails — Accurate at extreme percentiles; sketches merge without losing accuracy
Prometheus histogram — Counters per bucket, queried with histogram_quantile — Aggregates across instances; the estimate is only as fine as the buckets

> [!WARNING] > Never average p99 values from several servers. The mean of per-server p99s is not the p99 of all requests. Merge the histograms, then read the percentile.

Shortening the tail

Hedged requests — If a reply is late, send a second copy to another replica and use the first answer — Some duplicate work
Tied requests — Send to two servers at once; the first to start tells the other to cancel — Extra messages between servers
Timeouts from percentiles — Set timeouts just above a measured high percentile, not the average — Too tight causes retry storms
Fewer calls per request — Batch, cache or precompute so each request touches fewer servers — Design work, staler data
Less queueing — Keep headroom, isolate slow request types, prioritize interactive work — Idle capacity

Hedging is cheap when it waits. Sending the second request only after the first has taken longer than the 95th-percentile latency limits extra load to about 5%. In a Google benchmark reading 1,000 keys spread over 100 BigTable servers, hedging after a 10 ms delay cut the 99.9th-percentile latency from 1,800 ms to 74 ms, while sending just 2% more requests.

What you will be able to do

Define SLI, SLO, SLA and error budget, with an example of each.
Explain with numbers why an average hides slow requests.
Compute how fan-out turns a 1% slow rate into a majority of slow user requests.
Measure percentiles with mergeable histograms, and avoid averaging percentiles.
Choose among hedging, timeouts and design changes to shorten the tail.

Key concepts

SLI: Service level indicator: a measured number describing the service, such as p99 latency or success rate.
SLO: Service level objective: a target for an SLI, such as p99 under 300 ms over 28 days.
SLA: Service level agreement: a promise to users with consequences, such as credits, if the SLOs are missed.
Error budget: The share of requests an SLO allows to fail or be slow. It is spent by incidents and risky releases.
Percentile: The time within which a given share of requests finish. p99 means 99% are at least this fast.
Tail latency: The response times of the slowest requests, usually p99 and above.
Tail amplification: When a request waits for many parallel calls, rare slowness in each call becomes common slowness for the request.
Head-of-line blocking: Fast requests waiting behind slow ones in a queue, so their response time rises though their own work is quick.
Hedged request: A second copy of a late request sent to another replica; the first answer wins and the other is canceled.

When to use it

Setting latency and availability targets for a new service in a design review.
Explaining why dashboards and alerts track p99 rather than average latency.
Budgeting latency across a request that calls many services in parallel.
Choosing a timeout value or deciding whether to add hedged requests.

Interview questions

Why use p99 latency instead of the average for an SLO?
A page calls 50 backends in parallel, each slow 1% of the time. How often is the page slow, and what do you do?
How do you measure p99 latency across a fleet of 200 servers?
What is the difference between an SLO and an SLA, and how should they relate?