Distributed Tracing

How traces work, head vs tail sampling, and why traces are the highest-leverage signal for microservice debugging.

Overview

Where did the time go?

Distributed tracing answers a question metrics and logs can't: "this user request took 5 seconds; where did the time go across our 12 services?"

The mechanics

Trace — One user request crossing service boundaries
Span — One unit of work: one service handling part of the request, or one outbound RPC
Links — Spans share a trace_id; parent-child links via span_id and parent_span_id
Span contents — Name, start time, duration, status (OK / error), attributes such as http.method, db.statement, user.id

A trace is the tree of spans for one request. Drawn as a waterfall, each span's duration is a horizontal bar; the slowest span on the critical path is the bottleneck.

The hard part: sampling

Tracing every request in full is prohibitively expensive at scale: 10K req/sec × hundreds of attributes per span × tens of spans per trace is TB/day.

Head sampling — Decide at the gateway: "this trace is sampled at 10%" — Errors and slow requests are rare, so you miss what you most want to see
Tail sampling — Wait for the trace to finish; keep 100% of error and slow traces, 1% of normal — Needs a tail-sampling collector (OpenTelemetry tailsampling processor, or Honeycomb's sampler) and ~30s of span buffering

> [!TIP] 2026 best practice: tail sampling for error and slow traces plus head sampling for normal ones, or Honeycomb's similar "dynamic sampling."

OpenTelemetry tracing is ubiquitous

Auto-instrumentation packages (otel-instrumentation-http, otel-instrumentation-pg, and so on) create spans for HTTP, gRPC, database and message-broker calls with no code changes. Business-logic spans are added explicitly. Context propagates via W3C traceparent headers.

Good at, bad at

Latency investigations: exact breakdown of where time went — High-volume aggregation (use metrics)
Dependency mapping: who calls whom, in what order — Long-running processes: a 1-hour batch job has too many spans
Root cause for distributed failures: which downstream call failed — Anomaly detection across populations (use metrics or wide events)
N+1 detection: 50 sequential DB spans where one batched call would do

Key concepts

Trace: One user request's end-to-end journey across services. Identified by a single trace_id.
Span: One unit of work (a service's handling of part of a request, or an outbound call). Has name, start, duration, status, attributes.
Parent-child link: Spans form a tree via parent_span_id. The root span is the entry point; child spans are downstream work.
Head sampling: Decide at trace start (e.g., 10% of all requests). Easy but misses error / slow traces proportionally.
Tail sampling: Decide at trace end. 100% of errors / slow + 1% of normal. More useful, requires hold-and-decide infrastructure.
W3C traceparent: HTTP header standard for trace propagation. Every OTel SDK supports it; cross-vendor compatible.

When to use it

Latency investigations across services — find the slow span
Microservice dependency mapping — trace shows the actual graph, not the diagram
New-service rollouts — verify the request path matches your design
N+1 / chatty-pattern detection — sequential identical spans are the smell

Interview questions

How does head sampling differ from tail sampling — and which should I use?
What's the difference between a span attribute and a baggage item?
Distributed tracing has overhead. How do you minimise it?