Log Volume Explosion and Cost Runaway

Identify and eliminate runaway log volume that drives unexpected observability cost spikes.

Overview

Log volume explosions are one of the most common causes of unexpected cloud cost spikes, and they are almost always avoidable. One misconfigured service or a new code path logging at the wrong level can raise the observability bill 10x in hours, and most teams only notice when the monthly bill arrives.

The four usual causes

DEBUG logging in production — Logs every function call, query parameter and HTTP header; one misconfigured environment variable is enough — A service at 1,000 requests per second can emit tens of thousands of lines per second
Retry storms — Each retry is logged; if service A retries 5 times and calls B, which calls C, a failure in C multiplies lines across the chain — O(N^2) lines, spiking exactly during incidents
Verbose ORM query logging — Hibernate, SQLAlchemy and ActiveRecord print the full SQL for every query — Catastrophic at hundreds of queries per second
Health check logging — Load balancer and Kubernetes probes every 5-30 seconds, logged at INFO — 2-12 lines per minute per pod; significant across hundreds of pods

> [!WARNING] Retry storms spike log volume during incidents, burying the error logs you need in noise.

The cost

CloudWatch Logs ingestion — $0.50 per GB
One service at 100GB per day — $50 per day, $1,500 per month
Datadog log management — Similar pricing model

The fix: reduce at source, then filter in the pipeline

Dynamic log level API — An HTTP endpoint changes the log level without a restart
Sampling — Log 1% of successful health check responses, 100% of 5xx responses
Log routing — DEBUG to cheap cold storage (S3 via Firehose, $0.08/GB); ERROR to indexed storage (CloudWatch Logs, Datadog)

> [!TIP] Source reduction always comes first; pipeline filtering catches what slips through.

Key concepts

Dynamic Log Level: An HTTP endpoint or configuration mechanism that changes the log verbosity of a running service without restart, enabling DEBUG mode for specific investigations without a deployment cycle.
Log Sampling: Emitting only a fraction of log lines of a specific type — for example, logging 1% of successful health check responses — to reduce volume while preserving statistical visibility.
Log Routing: Directing log lines to different storage backends based on level or pattern — DEBUG to cheap cold storage, ERROR to indexed hot storage — to optimize cost vs. query speed.
Fluent Bit: A lightweight log processor and forwarder that can filter, transform, and route log streams using rewrite_tag and filter_grep plugins.
Log Ingestion Cost: The per-GB fee for writing logs to a managed service — CloudWatch Logs charges $0.50/GB ingested, with additional charges for storage and queries.

When to use it

Investigating an unexpected spike in CloudWatch Logs or Datadog log management costs
Diagnosing why a deployed service is generating more log volume than expected
Designing a log pipeline that balances observability with cost at scale
Explaining log sampling strategies that preserve signal while reducing volume
Setting up dynamic log level control for production services

Interview questions

How do you identify which log group or service is responsible for a log volume spike?
How do you implement log routing to reduce cost while preserving full ERROR log visibility?
What is a dynamic log level API and how does it work in practice?

Log Volume Explosion and Cost Runaway

Overview

Overview

The four usual causes

The cost

The fix: reduce at source, then filter in the pipeline

Key concepts

When to use it

Interview questions

More Troubleshooting topics