Three Pillars of Observability

What each pillar is for, where they fail, and the unified-via-OpenTelemetry shift.

Overview

Metrics, logs, traces

Observability is the property of a system that lets you understand its internal state from its external outputs. The practical decomposition has held for about 10 years: metrics, logs and traces, the "three pillars."

Each pillar's job

Metrics — Numerical, time-series, aggregated — Cheap — "How many requests per second? What's the p99 latency? How full is the disk?" — Prometheus, Mimir, Cortex, Datadog, CloudWatch
Logs — Structured or semi-structured events — Expensive (volume × retention) — "What did this specific request do? What was the stack trace?" — Loki, Elasticsearch/OpenSearch, Splunk, Datadog Logs
Traces — A tree of spans, each a unit of work — Expensive at full sampling — "This 5-second request: where did the time go?" — Jaeger, Tempo, Honeycomb, X-Ray

The 2020s shift: OpenTelemetry

Before OpenTelemetry (OTel), each pillar had its own SDK, wire format and collector. Now you instrument once and the OTel Collector routes each signal to its backend.

> [!NOTE] OTel is now CNCF's second-most-active project after Kubernetes.

The challenge to the pillars

Charity Majors, founder of Honeycomb, argues the pillars treat observability as three siloed datasets. Real debugging correlates across them: a metric spike, then the logs from that window, then a slow trace. Modern stacks (Honeycomb, Datadog, Grafana LGTM) build that correlation in.

> [!TIP] The three pillars are correct as a mental model but limited as an architecture.

Beyond the pillars

Profiles — Continuous CPU and memory profiles aggregated over time; find which function, not just which service — Pyroscope, Parca
Events — Deploy markers, feature-flag flips, scaling events, correlated against other signals

Key concepts

Metrics: Numerical, time-series, aggregated. Cheap. Drives alerts and dashboards. Counter, gauge, histogram, summary types.
Logs: Structured events. Expensive. Drives debugging and audit. JSON-formatted preferred over plain text for queryability.
Traces: Span trees showing causality across services. Drives latency investigations. Span = unit of work; trace = tree of spans sharing a trace_id.
OpenTelemetry (OTel): CNCF standard for instrumentation. One SDK emits metrics + logs + traces; OTel Collector routes to backends. The default for new services.
Continuous profiling: CPU/memory profiles aggregated over time (Pyroscope, Parca). Adds the "which function" dimension that traces and logs lack.
Correlation: The point of unified observability. Click from a metric spike into the logs from that range, into the slow trace within those logs.

When to use it

Designing observability for a new service — pick which pillars you need (usually all three)
Cost reviews — logs are usually the biggest spend; metrics are cheapest
"What's wrong with this service" investigations — start with metrics, drill into logs/traces
Picking a vendor or stack — open-source LGTM vs Datadog vs Honeycomb has different ergonomic and cost profiles

Interview questions

When do you reach for metrics vs logs vs traces?
What is OpenTelemetry and why does it matter?
What's wrong with the "three pillars" mental model?