Dashboards

Dashboard hierarchy (overview → service → resource), heatmaps for tail latency, and what NOT to put on a dashboard.

Overview

Dashboards are how humans consume metrics. Bad dashboards are noise; great ones orient on-call instantly.

1. Overview / executive — 5-15 panels, all SLO-derived; one user-facing SLO per panel ("Checkout availability 30d", "Search latency p99"); green/yellow/red against target — Leadership, engineering directors, on-call entry point
2. Service-level — One per service. Top row: Rate, Errors, Duration (or Traffic, Errors, Latency, Saturation). Middle: per-endpoint breakdowns. Bottom: dependency health (downstream latencies, DB query rates) — Service team, on-call SRE during incidents
3. Resource — Per host or resource, USE layout: CPU utilization and saturation, memory, disk, network — Infrastructure and platform team, deep debugging

> [!IMPORTANT] Drill down from overview to service to resource. Each level's job is to point you at the right view on the next level.

SLO-relevant metrics for the time window — Internal cause metrics that don't track user pain (random JVM gauges)
Latency histograms: heatmap with p50/p95/p99 lines — "Everything we have": link to detail dashboards instead
Recent deployment markers overlaid on timelines — Single-value gauges that should be alerts ("disk full!")
Counts grouped by relevant dimension (errors by endpoint, latency by route) — Stale metrics nobody looks at: delete, revive when needed

A line of "average latency" hides bimodality. A heatmap plots each request's latency (Y) against time (X), with color showing density.

> [!TIP] Grafana, Datadog and Honeycomb have heatmap widgets out of the box. Use one on every latency panel.

Alert overlays — Show fired alerts as bands or markers on the time axis. In review you see "alert X fired at 2:15pm" against the trend, saving correlation effort
Dashboard-as-code — Don't click-build in the UI. Check in Grafana JSON, the Datadog Terraform provider or Honeycomb's terraform module. Reviews catch bad metrics before they ship; rollback is git revert

Three-tier hierarchy: Overview (SLO-driven) → Service (RED/Golden) → Resource (USE). Each tier points to the next.
Heatmap: Latency density over time. Reveals bimodality that means / percentile lines hide. Standard widget in Grafana, Datadog, Honeycomb.
Deployment markers: Overlay deploy events on metric timelines. "Did the deploy at 2pm cause the latency spike at 2:01pm?" — visible at a glance.
Dashboard-as-code: Dashboards in git (JSON / Terraform). Reviewable, rollback-able, reusable across services.
Cardinality cost: Per-user or per-request dashboards mean huge cardinality on the metrics layer. Aggregate at the metric level; drill down via traces or logs.

Designing dashboards for a new service or product
Reviewing existing dashboards — are they still aligned with current SLOs?
On-call handover — does the dashboard set let a new on-call orient in <5 minutes?
Postmortems — could the dashboards have surfaced this incident faster?