Latency Diagnosis

Diagnosing network vs application latency, p99 vs average, cross-region trade-offs, and queuing theory.

Overview

Latency diagnosis requires distinguishing between different types of latency and their causes.

Network vs application latency

Network latency (propagation + transmission delay) is largely physics: ~1ms per 100km in fiber, ~70ms New York to London. Application latency adds queuing delay, processing time, and database round trips on top of network latency.

Percentile distributions matter

Average latency hides outliers. Use percentiles: - p50 — median; half of requests are faster - p99 — 1% of requests are slower than this value - p99.9 (tail latency) — often 5-10x p99; the worst users experience

In fan-out systems (one request calls 10 downstream services), the overall p99 approaches the p99.9 of each individual service. Reducing tail latency matters more than improving average latency in microservices.

Queuing theory — Little's Law

L = λW (L = items in system, λ = arrival rate, W = time per item). When arrival rate exceeds service rate, the queue grows unboundedly and latency spikes. This explains why a service at 80% CPU shows 2-3x latency vs 50% CPU — queuing effects are nonlinear near saturation.

Common latency sources (cloud)

DNS resolution — 5-50ms on first request to a new hostname
TCP handshake — ~1 RTT (3-way handshake)
TLS handshake — 1-2 RTTs (TLS 1.2) or 1 RTT (TLS 1.3)
Database queries — dominant for most OLTP applications
Cross-AZ traffic — 1-2ms within the same AWS region
Cross-region traffic — 40-200ms depending on geography

Key concepts

p99 latency: 99th percentile — 1% of requests are slower than this value. Use p99 and p99.9 for SLOs. Averages hide the long tail that users experience during spikes.
Queuing delay: Time waiting for the server to free up capacity. Grows nonlinearly as utilisation approaches 100%. At 90% CPU, queuing delay often exceeds actual processing time.
RTT (Round Trip Time): Time for a packet to travel from source to destination and back. Propagation delay is ~5ms per 1000km in fiber. TLS handshake adds 1-2 RTTs before data flows.
Connection keep-alive: Reusing TCP connections across HTTP requests eliminates the 3-way handshake and TLS renegotiation per request. Essential for high-throughput services.
Hedged requests: Sending a duplicate request to a second backend after a small timeout (e.g., 5ms), then using whichever responds first. Effectively eliminates tail latency at the cost of extra backend load (~1% overhead).

When to use it

Diagnosing a p99 latency spike that does not show up in the average
Explaining why a service at 70% CPU shows 3x more latency than at 30% CPU
Choosing the right AWS region pair to minimise cross-region latency for real-time features
Understanding why fan-out patterns amplify tail latency

Interview questions

Your API p99 latency is 2000ms but p50 is 50ms. What are the most likely causes and how do you diagnose?