Distributed Tracing

Trace requests across service boundaries. Spans, parent/child relationships, baggage, sampling. Tools: Jaeger (CNCF Graduated), Tempo (Grafana), Honeycomb (BubbleUp queries), Datadog APM. Solves the microservices "where is the latency?" problem.

Overview

Where is the latency?

Distributed tracing solves the microservices "where's the latency?" problem. A user sees 3-second checkout latency; without tracing you grep N services' logs hoping to correlate timestamps. With tracing you see the full request path as a flame graph, and the slow span jumps out.

> [!IMPORTANT] Every span in the picture shares one traceID; that shared ID is what turns N separate logs into one request story.

A short history

The 2020s consolidated on OpenTelemetry for the SDK and protocol layer (see the OpenTelemetry Fundamentals topic). Distributed tracing in 2026 means OTel-instrumented apps plus your choice of backend.

Backends compared

Jaeger — OSS; CNCF Graduated 2019; from Uber — Pure tracing backend; storage in Cassandra, OpenSearch or embedded Badger; self-host friendly — UI functional but dated; vendor free tiers (Honeycomb, Datadog) often beat it on UX
Tempo — OSS; Grafana Labs — Object-storage backed (S3, GCS), so cost scales with object storage, not memory or disk; fast lookup by traceID; excellent at scale — No span index by default; attribute search limited (TraceQL, 2023, partly solves it)
Honeycomb — Paid — BubbleUp compares an outlier trace's attributes to the population: "p99 traces in the last hour have user.tier=enterprise; the rest don't". Best-in-class high-cardinality debugging — Cost
Datadog APM — Paid — Polished UI, deep AWS and GCP integration, RUM + APM correlation; the most-used commercial tracing backend — Per-million-span billing at high RPS
New Relic — Paid — Comparable to Datadog; strong in the Java and .NET ecosystem — Cost

> [!TIP] OSS backends trail commercial ones on UX, especially for cardinality-heavy queries. The trade-off is cost: at high RPS, commercial backends bill aggressively per million spans.

Why tracing matters more than ever

Microservices proliferation — A 2026 e-commerce checkout typically crosses 15-30 services; the slow span is visible in seconds instead of hours of grep
LLM workloads — Sequential LLM calls (RAG retrieval, embedding, re-ranking, generation) each take 100ms-30s; the trace shows the whole chain and its hot spots
SaaS multi-tenancy — Traces tagged with tenant.id show "this trace from acme-corp is slow", bypassing average-of-averages metrics that hide per-tenant problems
Contract changes — When Service A's response shape changes and Service B breaks subtly, the trace shows the failing inter-service call directly

Three concepts every answer needs

1. Spans, traceIDs, parent-child — The data model: each span has traceID, spanID and parentSpanID; all spans of one trace share the traceID
2. Context propagation via W3C traceparent — How tracing crosses service boundaries; auto-instrumentation handles HTTP, gRPC and Kafka; custom protocols need manual propagation
3. Sampling and storage cost — Production produces billions of spans, so sampling is essential: head sampling at the SDK, tail sampling at the OTel Collector

Key concepts

Span anatomy — the tracing data model: A span represents one unit of work. Multiple spans form a trace via parent-child links. Span fields: Span kinds: - INTERNAL: in-process work. - SERVER: server-side request handling (HTTP server, gRPC server). - CLIENT: outbound call (HTTP client, DB query). - PRODUCER: message published to queue. - CONSUMER: message received from queue. Status: - UNSET (default): no explicit status. - OK: explicitly successful. - ERROR: failed; error attribute typically set. Trace tree relationships: - Parent/child: parentSpanID links to one parent span. Tree structure. - Links: one span can reference multiple other traces. Used for: batch processor span linking to upstream-individual-request traces; saga/workflow spans linking to step traces. Why this model: - Time-ordered: timestamps allow reconstruction of execution order. - Hierarchical: parent/child shows causality within a request. - Attributes: high-cardinality querying (find traces where user.id=X). - Events: discrete moments within a span (more lightweight than child spans). Span vs log line: a span has structure (kind, status, parent, duration). A log line is a single timestamp + message. Spans are richer; logs can attach to spans as events.
W3C Trace Context propagation: Standardized HTTP/gRPC headers for trace context. Ratified by W3C in 2020; supersedes vendor-specific formats (Datadog dd-trace-id, Jaeger uber-trace-id, B3 zipkin). Two headers: Format: {version}-{traceID}-{spanID}-{flags}. - version: 00 (current). - traceID: 32 hex chars (16 bytes; 128-bit ID). - spanID: 16 hex chars (8 bytes; 64-bit ID). - flags: 8 bits, currently only sampled (01 = sampled, 00 = not sampled). Vendor-specific extensions. Each vendor adds its key=value. Limited to 32 vendors. Propagation in practice (auto-instrumented HTTP server): Outbound (auto-instrumented HTTP client): Manual propagation for non-HTTP boundaries (async jobs, custom RPC): baggage (separate W3C spec): carries arbitrary key-value across service boundaries. Use case: pass tenant.id, user.id, feature flag values to every service in the chain without re-extracting from each request body. baggage values can be referenced in any span as attributes via SDK API. Common propagation gotchas: Non-instrumented middleware. nginx, envoy, AWS ALB don't propagate traceparent by default — incoming traceparent dropped at the proxy. Solutions: nginx propagation module, envoy native OTel filter, AWS X-Ray Daemon for ALB. Async boundaries. Producer creates trace; consumer is a different process. Manual context propagation required if message broker isn't auto-instrumented. Cross-region replication. Async replication breaks trace causality. Use links to reference the original trace; don't claim parent. Lambda / serverless cold starts. The init phase is hard to attribute. AWS Lambda's OTel integration creates a synthetic root span; subsequent invocations attach to it.
Sampling strategies in detail: Sampling is the cost-control mechanism for traces. Without it, billions of spans/day overwhelm storage. With it, you keep what matters. Strategy 1: Probabilistic head sampling at SDK. Behavior: at trace start, hash traceID → determine sample decision. Decision propagates via traceparent flags (01 sampled, 00 not). All services in trace agree. Pros: cheap. Spans dropped at SDK never hit Collector. Cons: random. Misses 99% of error traces if 1% sampling. Strategy 2: Tail sampling at OTel Collector. Decision after all spans of a trace arrive. Keep based on properties. Pros: informed. 100% of errors + slow + 1% baseline = ~5% retention with full signal. Cons: requires gateway pattern (load_balancing_exporter on agents to colocate spans). Memory-intensive. Strategy 3: Adaptive sampling. Honeycomb's Refinery: dynamic per-route sampling rates. High-traffic /health endpoint sampled at 0.1%; low-traffic /admin sampled at 100%. Maintains useful sample count per route regardless of traffic distribution. Datadog's adaptive sampling: similar behavior built into Datadog Agent. Refinery example config: Pros: high-cardinality services get appropriate sample rates. Cons: more configuration; less mature in OSS OTel. Strategy 4: Force-sample on rules. Override sampling for specific traces: Tail sampling policy: Used for: customer support reproducing user issues; debugging specific tenants. Sampling cost analysis: 100k RPS service, 10 spans/request, 30 days = 26B spans/month. - 100% retention: $2.6M/month at $0.10/M (Datadog APM). - 1% head: 260M spans, $26k/month. Random; misses 99% of errors. - Tail (5% effective): 1.3B spans, $130k/month. 100% errors + slow + 1% baseline. - Combined head (50% drop bots) + tail (5%): 650M spans, $65k/month. 100% errors + slow + 1% baseline. The combined approach gets you Datadog-scale insight at 25% of Datadog-scale cost. Production recommendations: Default: head sampling at 100% + tail sampling at gateway (errors + slow + 1% baseline). SDK overhead minimal at 100%; Collector does heavy lifting. High-traffic services: add SDK-level filtering for noise (health checks, bot UAs). Hot-path services: lower tail sampling baseline (0.1% instead of 1%). Multi-tenant: keep 100% of enterprise tier; sample lower tiers more aggressively. Critical: tail sampling decisions must be deterministic per traceID. Otherwise different services see different sampling decisions; trace fragments are missing.
Backend choice: Jaeger vs Tempo vs Honeycomb vs Datadog: Backends differ on storage cost, query capability, UX, vendor lock. Jaeger (CNCF Graduated, OSS). Architecture: collector + query + UI. Storage backends: Cassandra (production), OpenSearch (production), Badger (dev/single-node). Query model: by traceID (fast), by service+operation+tag (indexed). Full-text on attributes is expensive. UI: functional, dated. Flame graph view, service map, dependency analysis. Cost: self-host on K8s. Storage cost dominates: Cassandra cluster + ingestion infrastructure. Use when: OSS-only mandate; existing Cassandra investment; single-vendor-lock concern. Tempo (Grafana Labs, OSS). Architecture: distributor + ingester + querier. Storage: object storage (S3, GCS, Azure Blob, MinIO). Indexed by traceID; not by attributes (until TraceQL improvements). Query model: by traceID (fast). TraceQL (released 2023) adds attribute search but is slower than indexed alternatives. UI: via Grafana datasource. Trace flame graph; click span → related logs (Loki) and metrics (Mimir/Prometheus). Cost: object storage is cheap. Storage cost ~10x lower than Cassandra-backed Jaeger at scale. Use when: Grafana stack; cost-conscious at scale; query patterns dominated by traceID lookup (from logs/metrics correlation). Honeycomb (vendor, paid). Architecture: proprietary columnar store optimized for high-cardinality. Query model: BubbleUp — for any anomaly (slow trace, error trace), automatically compares anomalous trace's attributes against the rest of the population. Surfaces attributes that differ. "Why is this slow? p99 traces have user.region=eu-west-1; p50 don't." UI: best-in-class for high-cardinality debugging. Heatmaps, BubbleUp, custom queries. Cost: per-event pricing. ~$0.10-0.50 per million events depending on tier. Use when: high-cardinality apps (multi-tenant SaaS, ML inference); team values BubbleUp UX; budget allows vendor cost. Datadog APM (vendor, paid). Architecture: proprietary; tightly integrated with Datadog Logs + Metrics + Profiling + RUM. Query model: powerful trace search, service map, dependency graph. Distributed tracing tier integrates with metrics (Watchdog auto-anomaly detection). UI: polished. Single pane of glass for traces + logs + metrics + profiling. Cost: per-host + per-trace pricing. Aggressive at scale ($30-100k/month for moderate orgs). Use when: integrated DevOps stack; cost-tolerant; want polished UX. New Relic. Comparable to Datadog. Strong on Java/.NET. Per-data-ingest pricing model differs from per-host. Honeycomb vs Datadog (the two main commercial choices): Honeycomb: BubbleUp + high-cardinality. Better for multi-tenant SaaS, ML, debugging-heavy workflows. Datadog: integrated stack + polished UI. Better for general-purpose APM where cardinality is moderate. Pragmatic 2026 choice: OSS budget-conscious: Tempo + Grafana stack. Cheap; mature; Grafana ecosystem strong. OSS with rich querying: Jaeger if your team already runs it. New deployments: prefer Tempo. Multi-tenant SaaS / high cardinality: Honeycomb. Vendor-friendly, polished UX: Datadog or New Relic. All four accept OTLP from OTel-instrumented apps. Migration between backends is OTel Collector config change, not app re-instrumentation.
Recipe: Tempo + Grafana on EKS: Production deployment for an org running OTel + Tempo + Grafana. Step 1 — install Tempo via Helm: tempo-values.yaml: Step 2 — service exposed for OTel Collector: Step 3 — OTel Collector exports to Tempo: Step 4 — Grafana datasource: Step 5 — verify trace ingestion: Step 6 — query traces: In Grafana, Explore → Tempo → Search. Filter by service.name=payments, status=error. Click trace → flame graph. Click span → "View related logs" → Loki query for trace_id. Step 7 — TraceQL queries (Tempo 2.0+): More expressive than legacy tag filters. Released 2023; ongoing improvements through 2024-2026. Step 8 — service graph (auto-generated): Tempo's metricsGenerator auto-generates service-graph metrics from spans. Grafana renders the service dependency graph. Shows traffic between services, request rates, error rates. Cost considerations: S3 storage: ~$0.023/GB/month. 100GB/day of traces = ~$70/month for 90-day retention. Compute: distributors + ingesters + queriers ~5-15 vCPU + 10-30GB RAM total at moderate scale. ~$200-500/month EKS compute. Total: ~$300-1000/month for moderate-scale Tempo deployment. Compare to Datadog APM at same volume: $10-50k/month. ~30-100x cost difference. Operational considerations: Ingestion bursts: ingester replicas + retry_on_failure on Collector exporter handle. Tempo doesn't index spans by default. Search by attribute is slower than Jaeger/Honeycomb. TraceQL improving but still trails in 2026. For high-cardinality search (find all traces where user.id=X): Tempo can do it but Honeycomb's columnar store is faster. Backup: S3 lifecycle handles retention. No additional backup needed for traces (operational data; lose 30 days, no business impact).

When to use it

Microservices architectures with >5 services in any request path
Debugging cross-service latency / errors
LLM applications with multi-step pipelines
High-cardinality multi-tenant workloads
Replacing legacy log-only debugging — find slow spans in seconds vs hours

Interview questions

Walk through how a distributed trace gets created across 5 services.
How do you control distributed tracing costs at high RPS?
When is distributed tracing the wrong tool?
How do you handle async / queue-based tracing?
Quick-fire interview answers — distributed tracing essentials.