Cardinality & Observability Cost

Why cardinality blows up Prometheus, the math behind log/trace cost, and how to control them.

Overview

Observability is a major budget line

Observability is one of the largest line items in modern infrastructure budgets. Datadog's S-1 shows customers spending double-digit percentages of cloud spend on observability alone, so understanding the cost levers is now an SRE-grade skill.

Cardinality

Cardinality is the number of distinct label-value combinations for a metric.

5 services × 100 routes × 10 statuses × 5 methods — 25,000
Add user_id in a 100K-user system — 25,000 × 100,000 = 2.5 billion

> [!IMPORTANT] Labels are dimensions, not data. user_id is data; put it in logs or traces, which are event-shaped and naturally high-cardinality. Prometheus chokes on millions of series; billions are an OOM.

Cardinality bombs

The leading cause of Prometheus and observability outages:

Unbounded values: {path="/users/12345"} — Use the template: {route="/users/:id"}
User, session or request IDs, IP addresses as labels — Move to logs or traces
Free-form client labels — Sanitize them

The cost math per signal

Logs — Event size × event rate × retention — 2 KB/event (about 20 fields) × 5K events/sec × 86,400 sec/day — 864 GB/day raw; about 170 GB/day stored after about 5× gzip/zstd; 5 TB for 30 days hot
Traces — Span size × spans/trace × trace rate × sampling — 500 bytes/span (compressed) × 10 spans/trace × 1K traces/sec sampled at 10% — About 43 GB/day, about 300 GB for 7 days. At 100% sampling, about 430 GB/day
Metrics — Series × samples × retention — 10,000 series × 4 bytes/sample × 60 samples/min × 1,440 min/day — 3.5 GB/day raw, about 250 MB/day after Prometheus WAL and compression, 7.5 GB for 30 days

What that costs

Log ingest, 864 GB/day — About $1,725/day, roughly $52,000/month
Log storage, 5 TB — About $1,250/month

> [!WARNING] Metrics are cheap until cardinality explodes: grow to 10M series and you've multiplied by 1000×, about 250 GB/day. OOM.

The optimization stack

1. Bound cardinality at the SDK — Strip user_id from metric labels; keep it in logs and traces
2. Sample at the source — 1-10% of normal requests; 100% of errors and slow ones
3. Tier retention — Hot 7-14 days (queryable in seconds); cold 30-365 days on S3/GCS (queryable in minutes)
4. Drop high-cost fields you don't query — Full request bodies, debug fields, internal IDs
5. Pre-aggregate — A counter of "rate-limited events per user-hour" instead of one log per event
6. Per-service budgets — A $X/month ceiling per service; alert on a 2× spike

> [!TIP] The 2026 trend: nearly every team past 50 services runs a cost-optimization workstream. Common moves are leaving Datadog for LGTM (5-10× cheaper), aggressive tail sampling, retention tier-down and killing duplicate metrics.

Key concepts

Cardinality: Number of distinct label combinations on a metric. Linear in label distinct-values; multiplicative across labels.
Cardinality bomb: A label that takes unbounded values (user_id, request_id, IP) explodes the time-series count and OOMs Prometheus.
Sampling policy: Head sampling at gateway (% of all traces) + tail sampling at collector (% by error / latency). Standard 1-10% + 100% errors.
Retention tiering: Hot (queryable, days) + cold (archived, months/years). Loki, Tempo, Mimir support tiered storage natively.
Pre-aggregation: Convert per-event logs to per-time-window metrics. "rate_limited per user per hour" instead of per-event log.

When to use it

Designing observability for a new service — set cardinality and sampling policy upfront
Cost-reduction projects — almost every observability cost optimization starts with cardinality and sampling audit
Capacity planning — Prometheus / Tempo / Loki sizing depends on these numbers
Vendor evaluation — pricing models map to these dimensions

Interview questions

I added a metric label and Prometheus is OOMing. Why?
How do you decide on a trace sampling rate?
How would you cut a $200K/month Datadog bill?