Log Aggregation

Centralized logs across services. Three storage models: full-text indexed (Elasticsearch / OpenSearch), label-indexed (Loki, ~10x cheaper), columnar (ClickHouse, hyperscale). Collection layer: Fluent Bit / Vector / Promtail / OpenTelemetry. Cost vs query power trade-off.

Overview

Logs: the most detail, at the highest cost

Logs are the third leg of observability after metrics and traces. They carry the most semantic detail per event, at the highest cost per useful query. Modern log aggregation balances cost (storage and indexing) against query capability.

The 2026 landscape

Elastic Stack / OpenSearch — Elasticsearch + Logstash + Kibana (ELK) dominated log search ~2015-2021 with strong full-text search. The 2021 license shift (Elastic License v2, not OSS-compatible) led AWS to fork OpenSearch (Apache 2.0) — Elastic if vendor-friendly, OpenSearch if OSS-friendly; both production-ready
Grafana Loki — Released 2018 by Grafana Labs; designed to be cheap by indexing labels only, never log content; ~10x cheaper than Elastic at scale — Default for cost-conscious cloud-native shops
ClickHouse — OLAP database used for logs at hyperscale (Uber, Cloudflare, ByteDance); columnar, compression-friendly, SQL — PB/day scale; steep learning curve
Cloud SaaS — Datadog Logs, Splunk Cloud, New Relic Logs; vendor-managed, pricing scales aggressively with volume — Dominant in mid-size shops; expensive at scale

Forwarders

Fluent Bit — Default cloud-native forwarder; CNCF Graduated as part of the Fluentd project
Vector — Rust, high throughput, best transformation language
Promtail — Grafana's, optimized for Loki
OTel Collector — Unified for traces, metrics and logs

Why aggregate logs at all

Cross-service debugging — Single-service stdout does not show distributed flow; central logs with trace_id correlation reveal cross-service issues
Compliance retention — SOC 2, HIPAA and PCI mandate retention (90d-7y); container logs are ephemeral, central logs are durable
Search at scale — Grep across 100 services without distributed shell access
Anomaly detection — A spike in logged 500s triggers an alert; pattern detection across volume
Forensics — What happened in the 30 minutes before the outage?

Where logging fails

Cost per observation — Each line is more verbose than a metric data point; storage costs 100-1000x metrics for the same observation rate
Latency — Typically 1-30s ingestion delay; not real-time
High-volume queries — "Find all logs where user.id = 42" across petabytes is expensive on any backend

> [!WARNING] Logs are the most expensive signal per event. Use metrics for counts and trends, and logs for detail.

Three concepts every answer needs

1. Three storage models — Full-text indexed (Elasticsearch, OpenSearch), label-indexed (Loki), columnar (ClickHouse): a cost vs query-capability trade-off
2. Forwarder layer — Fluent Bit, Vector, Promtail or the OTel Collector parse, enrich and ship logs; on K8s they run as a DaemonSet
3. Cardinality vs index size — Loki: under 10k label streams per cluster. Elastic: index size 50-200% of raw logs

Key concepts

Loki vs Elasticsearch — cost and query trade-offs: The defining choice for OSS log aggregation in 2026. Elasticsearch / OpenSearch: full-text indexed. Every log line tokenized; inverted index built. Query "service:payments AND error" → instant. Cost structure: - Storage: index is 50-200% of raw log size. 1TB raw = 1.5-3TB on disk. - Memory: indexes loaded into RAM for fast queries. ~25% of disk size in RAM. - Compute: index merges, refreshes, reindexes — CPU-heavy. For 1TB/day raw logs, 30-day retention: - Disk: 60-90TB. - RAM: 15-23TB across the cluster. - Compute: ~50-100 vCPU for moderate query load. - Cost (Elastic Cloud or self-hosted EBS): $5k-15k/month. Strengths: - Best-in-class full-text search. KQL / Lucene queries are powerful. - Mature aggregation framework. Date histograms, percentiles, derivatives. - Kibana UI is polished. Weaknesses: - Storage cost dominant. - License: Elastic moved to non-OSS license in 2021. OpenSearch (AWS fork) is Apache 2.0; new deployments increasingly choose OpenSearch. - Cardinality of fields impacts index size; field explosion is a real concern. Loki: label-indexed. Index small label set (cluster, namespace, service, level). Log content stored as compressed chunks. Query mechanism: 1. Index narrows by labels: {service="payments"} → list of chunks. 2. Querier fetches matching chunks from S3. 3. Grep through chunks for content match. Cost structure: - Storage: chunks compressed ~10x. 1TB raw = ~100GB compressed. - Object storage: S3/GCS/Azure Blob. ~$0.023/GB/month. - Compute: ingesters + queriers — moderate. For 1TB/day raw logs, 30-day retention: - S3: 3TB. ~$70/month. - Compute: 5-10 vCPU. ~$300-500/month. - Cost: ~$400-600/month total. Strengths: - ~10x cheaper than Elasticsearch at scale. - Native Grafana integration. - Cardinality-aware: forces good label hygiene; punishes high-cardinality at usage time. Weaknesses: - Full-text search without labels = full chunk scan. Slow. - Schema flexibility: structured logs (JSON) parse with | json operator; works but less ergonomic than ES. - Query language (LogQL) less expressive than KQL/Lucene for complex full-text patterns. When to pick Elasticsearch / OpenSearch: - Full-text search is the primary use case. - Volume 100GB/day. - Already on Grafana stack. - K8s-native shop. - Most queries narrow by service/namespace/cluster (label hits effective). When to pick ClickHouse: - Hyperscale (>10TB/day). - Analytical queries (group by, aggregations) over log data. - Have ClickHouse expertise. Many large orgs run hybrid: Loki for high-volume logs (web, infra); Elasticsearch for low-volume but rich-search (audit, security).
LogQL — Loki query language: LogQL borrows PromQL syntax for logs. Two query types: log queries (return logs) and metric queries (aggregate logs as metrics). Log queries: Parse structured logs: Metric queries (transform logs to metrics): Use case: alert on log-derived metrics. Without separate metrics infrastructure, derive "error log rate" from logs alone. Recording rules in Loki (Loki ruler component): Pre-computed; queryable as metrics in Grafana via Loki datasource. Common patterns: Find unique values: Top N error sources: Find slow requests: Trace ID lookup (cross-correlate with Tempo): Limitations vs PromQL: aggregations less powerful (no histogram_quantile equivalent on log data; would need bucket-emit pattern). Streaming queries (live tail) supported. LogQL gotchas: - |= is contains (substring match); fast. - |~ is regex; slower. - | json parses every line; slow if not pre-filtered by labels. - Always filter by labels first; content filters/parsers next.
Forwarder choice — Fluent Bit, Vector, Promtail, OTel Collector: Forwarders read logs from sources, transform, ship to backends. Four common choices in 2026. Fluent Bit. CNCF Graduated 2024. C-based; ~5MB memory footprint. Default for cloud-native. Strengths: tiny memory, fast, C-based reliability. Standard for K8s DaemonSet log forwarder. Outputs to Loki, Elasticsearch, S3, Kafka, OTLP, custom. Weaknesses: simple filter language; complex transformations awkward. Vector. Rust-based. By Datadog (acquired Timber.io 2021); OSS Apache 2.0. Strengths: VRL (Vector Remap Language) is the best transformation language in the space. Multi-pipeline routing (one source → many sinks). Higher throughput than Fluent Bit. Weaknesses: ~50MB memory (10x Fluent Bit). Newer; smaller community. Promtail. Grafana Labs. Tightly integrated with Loki. Strengths: native Loki integration; automatic K8s service discovery; pipeline stages for parse/transform. Weaknesses: Loki-specific (won't ship to Elastic); less actively developed than Vector. OpenTelemetry Collector. Unified for traces+metrics+logs. Strengths: one Collector for all three signals; standardized config; future-proof (OTel ecosystem). Weaknesses: heavier than Fluent Bit (~30MB); transformation language less rich than Vector's VRL. Decision matrix: - Default cloud-native, tight memory budget: Fluent Bit. - Complex transformations / multi-sink routing: Vector. - Loki-only, Grafana-stack: Promtail. - Already deploying OTel Collector for traces/metrics: extend to logs. Hybrid common: Fluent Bit per node for pure forwarding; Vector at gateway for heavy transformation; both pushing to Loki.
Structured logging best practices: Logs are most valuable when structured (JSON or logfmt). Unstructured plain-text logs become unsearchable at scale. Bad (unstructured): Good (JSON): Why structured wins: - Searchable: filter by user_id=42, error_type=card_declined. - Aggregatable: rate(error_type=card_declined) over time. - Correlatable: trace_id links to distributed trace. - Type-safe: amount is a number, not a string to parse. Library support: Python: Node.js: Go: Java: Standard fields (semantic conventions): - timestamp: ISO 8601. - level: trace, debug, info, warn, error, fatal (lowercase). - service: app/service name. - message: human-readable summary. - trace_id, span_id: from active OTel span. - error.type, error.stack: for error logs. - http.method, http.path, http.status_code: for request logs. OTel semantic conventions for logs (catching up to traces' maturity in 2024+) standardize these field names. Anti-patterns: 1. Logging full request body. Captures PII, blows up storage. Log only metadata (size, user_id, route). 2. Logging in tight loops. Per-row logs in a 1M-row processing job → 1M log lines per job. Aggregate at job level instead. 3. Multi-line stack traces without proper handling. Forwarders may treat each line as separate record. Use multi-line parsers (Fluent Bit's multiline filter). 4. Mixing log formats in same service. JSON in some places, plain text in others. Backend can't parse consistently. 5. Sensitive data: passwords, tokens, full credit card numbers, API keys. Redact at forwarder level. Never trust app-level redaction alone. PII redaction at forwarder: Vector's VRL: Fluent Bit's modify filter: Log levels in production: - error / fatal: actually erroneous; alert-worthy. - warn: degraded but functional; review. - info: significant business events (user registered, order placed). Production volume. - debug: development only; disable in prod (reduces volume 10-100x). trace level: never in prod; very chatty. Sampling for high-volume info logs: If info logs are too voluminous, sample at forwarder (1 in 10 lines kept). Loses fidelity; saves cost. Reserve full sampling for error/warn levels.
Recipe: Loki + Promtail on EKS with multi-tenant retention: Production Loki deployment. Step 1 — install Loki via Helm chart (loki-distributed): Step 2 — Promtail DaemonSet: Step 3 — multi-tenant per-team config: Tenants identified via X-Scope-OrgID header on push and query. Step 4 — Grafana datasource: derivedFields: in Loki query results, render trace_id as link to Tempo. Click → trace flame graph. Step 5 — alert rules via PrometheusRule (Loki Ruler): Step 6 — observability for Loki itself. Loki emits Prometheus metrics on /metrics. ServiceMonitor for kube-prometheus-stack. Key metrics: - loki_distributor_lines_received_total (ingest rate) - loki_ingester_chunks_flushed_total (chunk flush rate) - loki_request_duration_seconds (query latency) - loki_logql_querystats_latency_seconds_sum (LogQL query perf) Cost analysis: 100 services, 50KB/sec average log volume, 30-day retention = 130GB/day raw, 13GB/day compressed. S3: 13GB × 30 = 390GB. ~$10/month. Compute: 5-10 vCPU + 20Gi memory. ~$300-500/month. Total: ~$400/month moderate scale. Compare Datadog Logs at this volume: $5k-15k/month. Loki ~30x cheaper. Operational: Loki is K8s-native; runs alongside Prometheus/Tempo on same cluster; shared object storage.

When to use it

Multi-service architectures where centralized debugging matters
Long retention for compliance (SOC 2, HIPAA, PCI)
High-volume logs where cost of storage matters (Loki saves 10x vs Elastic)
Replacing distributed grep across N services
Anomaly detection on log patterns

Interview questions

How do you choose between Loki and Elasticsearch?
Walk through structured logging best practices and PII handling.
How do you correlate logs with traces and metrics?
How do you control log volume and cost at scale?
Quick-fire interview answers — log aggregation essentials.