Logging Best Practices

Structured JSON over text, log levels, sampling, correlation IDs, and the cost reality.

Overview

The most expensive pillar

Logging is the most expensive of the three pillars at scale and the most often abused. Two engineers logging "everything at INFO" can quintuple a service's log spend without proportional debug value.

Six rules that hold up in production

1 — Structured (JSON) over plain text — Plain text needs regex to query; structured logs are queryable as data. Loki, Elasticsearch, CloudWatch and Datadog treat JSON natively
2 — Calibrate levels — "Log everything at INFO" is the mistake; INFO should be 1-3 lines per request, not 50
3 — Sample at the source for high-volume services — 100K req/sec is 8.6 billion events/day. Sample 1-10% by trace context; keep 100% of errors and slow requests. OpenTelemetry and Sentry support this natively
4 — Correlation IDs everywhere — Every line carries the request's trace_id, so trace_id="abc" correlates logs, traces and metrics (via exemplars). Without it, you grep timestamps and pray
5 — Don't log secrets — PII, tokens, passwords, full request bodies. A leak costs far more than lost debug context. Use redaction libraries; allowlist what to log rather than blocklist what to omit
6 — Retention is a budget knob — Hot (queryable) 7-30 days; cold archive (S3/GCS) for compliance and forensics. Loki + S3 makes this cheap; Datadog's default 15-day hot retention adds up

Structured beats plain text

This line: {"ts":"2026-05-04T12:00:00Z","level":"warn","service":"api","trace_id":"abc","user_id":12345,"event":"rate_limited","limit":100} beats: 2026-05-04 12:00:00 WARN api: user 12345 rate-limited (limit=100)

A canonical level scale

TRACE — Hot-path step-by-step; rarely enabled in prod
DEBUG — Useful for debugging but noisy; off by default in prod
INFO — Discrete events of interest (request handled, job started)
WARN — Handled but suspicious (retry, fallback, deprecated path)
ERROR — Handled but bad (caught exception, rejected request)
FATAL — Crashing now

Cost reality

Volume — 1 KB/entry × 100 req/sec × 10 lines/request × 86,400 sec/day = about 86 GB/day
30-day hot retention — About 2.5 TB
Datadog pricing — Around $1.7-$3/GB ingested, plus storage
One moderately busy service — $4K-$8K/month

> [!WARNING] Multiply by 50 services and you see why log cost dominates observability budgets.

Key concepts

Structured logging: JSON or key-value lines instead of plain text. Queryable as data. Standard in 2026.
Log levels: TRACE / DEBUG / INFO / WARN / ERROR / FATAL. Calibrate so INFO is rare; "everything at INFO" is a smell.
Sampling at source: 1-10% of requests for high-volume services; 100% for errors / slow requests. Decide at the SDK, not the backend.
Correlation ID (trace_id): Propagated through every log line in a request. Lets you join logs to traces and metrics. OTel propagates automatically.
Log retention tiering: Hot (queryable, days-weeks) + cold (archived, months-years). Major cost lever; tune per service.

When to use it

Designing logging strategy for a new service — set the levels, structured format, and sampling rate up front
Cost reviews — logs are usually the biggest line item; sampling/retention are the levers
Incident-response gaps — "we couldn't find the right log" is fixed by structure + correlation IDs
Compliance / audit — separate audit-log path with longer retention, no sampling

Interview questions

What's a good logging policy for a new service?
How do correlation IDs work in distributed systems?
How do you reduce log costs without losing visibility?