Missing Traces in Distributed Tracing

Overview

Overview

Distributed tracing is only valuable when traces are complete. A trace that shows three of seven service hops gives misleading latency attribution and can send you debugging the wrong service; the good news is that missing traces have a small number of root causes.

The four root causes

• Sampling — Head-based sampling decides at the entry point, typically 1% to 10% of traffic — Rare errors (0.1% of requests) and slow outliers are almost never captured
• Broken context propagation — A hop fails to forward the trace ID, span ID and sampling flag — The downstream service starts a new root span with a new trace ID
• Instrumentation gap — A service has no tracing library, so it cannot create spans or forward context — Span A completes, span C starts much later, no span B in between
• Exporter misconfiguration — Wrong endpoint, TLS failure, wrong port or a saturated batch queue — The app runs, tracing looks configured, nothing reaches the backend

Propagation headers

• W3C Trace Context — traceparent
• Zipkin B3 — X-B3-TraceId and related headers

> [!WARNING] Load balancers, API gateways, AWS Lambda event sources and background queue consumers are the usual places the chain breaks.

Checks and what each tells you

• OTel Collector zPages /tracez — Span counts, latency distributions and error rates for spans through the collector
• Exporter logs — Connection errors to the backend
• Span counts over time — Whether spans are still arriving

Sampling: head versus tail

• Decision point — Entry point, before the trace completes — Collector, after buffering the complete trace
• Keeps errors and outliers — Rarely — Yes, decided on trace characteristics
• Requirement — None — Stateful collector, such as the OpenTelemetry Collector tail sampling processor

> [!TIP] If you are debugging a specific error and its trace is missing, suspect sampling first.

Key concepts

Head Sampling

A sampling strategy where the decision to trace a request is made at the entry point before the trace completes — fast and stateless but misses rare events.

Tail Sampling

A sampling strategy where a stateful collector buffers complete traces and makes the sampling decision based on trace characteristics like errors or high latency.

Context Propagation

The mechanism for passing trace context (trace ID, span ID, sampling flag) between services via HTTP headers or message metadata.

traceparent

The W3C Trace Context standard HTTP header carrying trace ID, parent span ID, and trace flags — the modern standard for context propagation.

zPages

A debug endpoint built into the OpenTelemetry Collector that exposes span counts, latency distributions, and errors for in-flight tracing data.

When to use it

• Diagnosing incomplete waterfall diagrams in Jaeger or Zipkin
• Explaining why traces for error requests are never captured despite tracing being enabled
• Debugging context propagation breaks at API gateways, load balancers, or message queues
• Choosing between head sampling and tail sampling for a production tracing deployment
• Investigating why spans are generated in the application but never appear in the tracing backend

Interview questions

• What is the difference between head sampling and tail sampling, and when should you use each?
• How do you debug context propagation breaks in a microservices tracing setup?
• What are the most common reasons spans are generated by the application but never appear in Jaeger or Zipkin?

More Troubleshooting topics

• JVM Garbage Collection Pauses
• Kubernetes Service Not Reachable
• Lambda Cold Start Latency
• Log Volume Explosion and Cost Runaway
• Memory Leak Diagnosis
• Metrics Gaps in Dashboards
• Multi-Region High Latency
• NAT Gateway Bandwidth and Throttling
• Network Bandwidth Saturation
• OOMKilled Pod Diagnosis
• Packet Loss and Network Degradation
• Pods Stuck Pending