Prometheus & Grafana

CNCF Graduated metrics stack. Prometheus pulls metrics on intervals, stores TSDB locally. Grafana queries via PromQL. Mimir / Thanos / VictoriaMetrics for HA + long-term storage. Replaced Graphite, Nagios, Sensu in modern stacks.

Overview

The default metrics stack on Kubernetes

Prometheus is the dominant metrics monitoring system in cloud-native, and Grafana is the dominant visualization layer on top of it. Together they replaced Graphite, Nagios, Sensu and Zabbix in most cloud-native deployments; by 2026 they are the default metrics stack on Kubernetes.

Prometheus — Originated at SoundCloud (2012), inspired by Google's Borgmon; CNCF Graduated 2018, the second project after Kubernetes. Pull-based model, time-series database, PromQL query language
Grafana — First released 2014 by what became Grafana Labs; not a CNCF project. Open source (AGPLv3 since 2021, Apache 2.0 before), with Grafana Cloud as the paid managed tier. Visualizes Prometheus, Loki, Tempo, InfluxDB, Datadog and dozens of other data sources

Why Prometheus wins

Pull-based simplicity — Prometheus scrapes /metrics; apps just expose the endpoint, with no client push complexity. If Prometheus cannot reach the app, that is itself a metric (up=0)
Service discovery — Native K8s integration (kubernetes_sd_config) auto-discovers pods and services with the prometheus.io/scrape annotation; a new deployment is monitored automatically
PromQL — Aggregations (sum, avg, max), rate calculations, histogram quantiles, predictions; the de facto standard, supported by Mimir, Thanos and VictoriaMetrics
Local TSDB — Embedded time-series database, so no external DB for small deployments; compressed columnar storage; tens of millions of series per node
Standard format — Text-based exposition format, standardized by OpenMetrics (CNCF); hundreds of client libraries and thousands of exporters (mysqld_exporter, postgres_exporter, redis_exporter, blackbox_exporter for HTTP probes)

Where Prometheus has friction

Single-node by default — HA means two Prometheus instances scraping the same targets, deduplicated by Mimir or Thanos; operational overhead
Limited retention — 15-day default; longer retention slows queries, so you need long-term storage (Mimir, Thanos)
High cardinality kills — Each unique label combination is one time series and storage cost is per series; a label with millions of values (user.id) explodes the index
Pull model limits — Short-lived jobs (batch, Lambda) need Pushgateway, a proxy that receives pushes and serves pulls; more operational complexity

> [!WARNING] Cardinality, not sample volume, is what takes Prometheus down. Never put unbounded values such as user IDs in labels.

When to pick it

Greenfield K8s monitoring: the default in 2026 — Tiny apps with no K8s: a hosted service (Datadog, Grafana Cloud) for zero ops
Cost-conscious at scale: OSS Prometheus + Mimir + Grafana is 10-50x cheaper than Datadog or New Relic for high-volume metrics — Push-only metrics (Lambda, batch jobs): Pushgateway works but is awkward; managed services handle it better
OpenMetrics-friendly ecosystem: Kubernetes, Argo CD, Flux, Istio, Envoy, etcd and all major DBs expose /metrics natively — Need polished commercial UX: Datadog and New Relic dashboards beat hand-rolled Grafana

Three concepts every answer needs

1. Pull-based /metrics scraping — Apps expose; Prometheus scrapes; no client push
2. PromQL aggregations — rate() for counters, histogram_quantile() for percentiles, sum() by () for grouped aggregation
3. Long-term storage tier — Mimir, Thanos or VictoriaMetrics for HA and retention beyond local Prometheus's limits

Key concepts

Metric types — counter, gauge, histogram, summary: Four metric types in Prometheus. Choose based on what you measure. Counter: monotonically increasing. Reset only on process restart. Use for: request count, error count, bytes received. Query: rate() converts counter to per-second rate. - rate(http_requests_total[5m]) → req/sec averaged over last 5 minutes. - increase(http_requests_total[1h]) → total over last hour. Never sum counters across instances directly without rate(); accumulating raw counter values is meaningless. Gauge: arbitrary up-down value. Use for: current memory usage, queue depth, in-flight requests. Query: directly. avg(queue_depth) by (queue_name) gives current depth per queue. Histogram: distribution of values via cumulative buckets. Use for: latency, response size, batch size. Histogram emits multiple time series: - *_bucket (cumulative count per bucket). - *_sum (sum of all values). - *_count (count of all observations). Query: histogram_quantile() computes percentiles. - histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) → p99 latency. - histogram_quantile(0.5, ...) → median (p50). Bucket selection matters: too few buckets → coarse percentiles; too many → high cardinality. Summary: similar to histogram but pre-computes quantiles client-side. Emits: *_sum, *_count, *_count{quantile="0.5"}, *_count{quantile="0.99"}. Trade-off vs histogram: - Summary: client computes quantiles. Accurate per-instance percentiles. CAN'T aggregate quantiles across instances (taking p99 of p99 is mathematically wrong). - Histogram: server computes quantiles from buckets. Slightly approximate but CAN aggregate across instances correctly. Best practice: use histogram unless you have specific reason for summary. Aggregation matters more than per-instance accuracy. Common label sets: - HTTP: method, route, status_code. - DB: db_system, operation. - Queue: queue_name, status. Avoid high-cardinality labels (user_id, full URL, error message) — each unique value = new time series.
PromQL essentials: PromQL is the query language. Two query types: instant (single point in time) and range (over a time range). Selectors: Range vector selectors: append [duration]: Functions on counters: rate() for graphs (smooth); irate() for alerts (responsive); increase() for "how many in window". Aggregations: Label manipulation: Histogram quantiles: Critical: include le (less-than-or-equal-to bucket boundary) in by clause. Otherwise quantile calculation breaks. Recording rules pre-compute: Subsequent queries: Pre-computed; queries are fast. Use for dashboards and alerts that hit expensive aggregations. Alert rules: for: 5m means: alert fires only after the condition has been true for 5 minutes. Avoids flapping. Common patterns: CPU saturation: Memory pressure: Pod restarts: Disk space prediction (4 hours forward):
Service discovery and exporters: Prometheus discovers targets dynamically. Six common discovery types: kubernetes_sd_config (most common in K8s): Pod with annotation prometheus.io/scrape: 'true' is auto-discovered. relabel_configs filter and transform discovered targets. ServiceMonitor / PodMonitor (kube-prometheus-stack CRDs): Higher-level abstraction. ServiceMonitor in tenant namespace; Prometheus Operator picks it up. Other discovery: consul_sd, ec2_sd_config, gce_sd_config, azure_sd_config, file_sd (refresh from JSON file). Static config (non-K8s): Exporters expose metrics for systems that don't natively speak Prometheus: Common exporters: - node_exporter — Linux host metrics (CPU, memory, disk, network). - kube-state-metrics — K8s object state (pods, deployments, replicasets). - cadvisor — container resource usage (built into kubelet). - mysqld_exporter, postgres_exporter, redis_exporter, mongodb_exporter — DB metrics. - blackbox_exporter — HTTP/HTTPS/TCP/ICMP probes for synthetic monitoring. - snmp_exporter — SNMP-based network device monitoring. - jmx_exporter — JVM metrics. - statsd_exporter — bridge from StatsD push to Prometheus pull. Exporters run as sidecars or DaemonSets, expose /metrics, Prometheus scrapes them. Recording rules at exporter time vs Prometheus time: Exporter metrics are raw counters. Recording rules in Prometheus pre-aggregate (rate, sum). Heavy aggregation = recording rule; light query = on-demand PromQL. Production pattern: raw counter exporters → recording rules → dashboard PromQL queries. Three layers; performant.
Alertmanager — routing, grouping, silencing: Alertmanager handles alerts post-Prometheus. Prometheus's evaluator decides "this is an alert"; Alertmanager decides "what to do with it". Routing tree: Grouping: alertname + cluster + service grouped together. Many similar alerts → one notification batch. Inhibition: suppress lower-severity alerts when higher-severity is firing: If "API down" critical alert fires, suppresses "API slow" warning for the same service. Silences: temporary suppress alerts during planned maintenance: Or via UI. Silenced alerts don't fire to receivers but remain visible in Alertmanager. Modern alternative: Grafana Alerting (Grafana 10+). Replaces Alertmanager for many users: - Unified UI in Grafana for alert management. - Multi-data-source alerts (combine PromQL + LogQL + TraceQL). - Better visualization. Prometheus → Grafana Alerting → notification channels. Some teams keep Alertmanager for routing; others go fully on Grafana Alerting. Alert hygiene best practices: - Every alert: clear runbook URL in annotations. - for: duration to avoid flapping (5m for slow conditions, 30s for hard failures). - severity labels: page only critical; chat for warnings; email/no-page for info. - SLO-based alerts (multi-burn-rate) over threshold-based where possible. - Alert on symptoms (errors, latency), not causes (CPU, memory) where possible.
Recipe: kube-prometheus-stack on EKS: Production-grade Prometheus deployment via Helm. Step 1 — install via Helm: values.yaml: Step 2 — verify pods come up: Step 3 — apps expose ServiceMonitor: Step 4 — recording rules + alert rules via PrometheusRule CRD: Step 5 — Grafana dashboards. Pre-built dashboards from grafana.com/dashboards (Kubernetes Cluster Monitoring, Node Exporter, etc.). Import via dashboard ID. Custom dashboards: build in Grafana UI; export JSON; commit to git; provision via grafana-dashboard ConfigMap. Operational notes: HA: Prometheus replicas = 2; both scrape same targets independently; Mimir/Thanos deduplicate. Lose one Prometheus → other continues. Cost: ~5-10 vCPU + 30-50Gi memory + 200-500Gi disk for moderate cluster (50 services). ~$300-700/month EKS compute + storage. Backup: Prometheus's local TSDB is mostly disposable (rebuilt from scrapes). Long-term data in Mimir's S3 is the durable layer. No additional backup needed for Prometheus. Upgrade: Helm chart upgrade. CRD changes occasionally; test in staging first.

When to use it

K8s metrics monitoring — default 2026 stack
Cost-conscious at scale (OSS vs Datadog/New Relic)
OpenMetrics-friendly cloud-native apps
Multi-tenant clusters (Mimir x-scope-orgid)
Long retention required (years of metrics; archive analysis)

Interview questions

Why pull-based instead of push-based metrics?
How do you handle high cardinality and storage cost?
Walk through migrating from Datadog to Prometheus + Grafana.
How do you build effective alerts in Prometheus?
Quick-fire interview answers — Prometheus + Grafana essentials.