Metrics Gaps in Dashboards

Overview

Overview

A gap in a dashboard graph is not the same thing as a zero value. Treating them as equivalent is a common diagnostic mistake that wastes hours of on-call time, so the first step is knowing what actually causes a gap.

The four Prometheus causes

• Scrape failure — Prometheus gets a connection error, timeout or non-200 response and writes a staleness marker, a sentinel meaning "this series has gone away" — Broken line in Grafana, not a flat zero
• Network partition — Packet loss makes some scrape cycles fail while others succeed — Intermittent gaps, common in cross-VPC or cross-AZ setups with misconfigured route tables
• Clock skew — Samples arrive out of order and the storage engine silently drops them — Gap in Grafana but not in the raw /metrics output on the target
• Label change — A pod restarts with a new uid label or an ASG replaces an instance, so the old label set disappears — Queries that hardcode old label values show a gap while the metric is scraped fine

> [!NOTE] Clock skew only drops samples when the target exposes its own timestamps and its clock runs more than a few seconds ahead of the Prometheus server; otherwise Prometheus stamps samples with its own scrape time.

Checks and what each tells you

• up{job="myjob"} == 0 — The target is discovered but the scrape is failing
• No result for up{...} — The target is not in Prometheus's discovery output
• Prometheus /targets page (port 9090) — Every discovered target, its last scrape time and the scrape error
• scrape_duration_seconds near scrape_timeout (default 10s) — Scrapes are timing out intermittently
• chronyc tracking or timedatectl on both hosts — Whether the clocks are skewed

CloudWatch gaps

In CloudWatch, gaps appear when the CloudWatch agent stops sending metrics: on instance termination, agent crashes, or network ACL changes that block outbound traffic to the CloudWatch endpoint.

> [!TIP] Use FILL(m1, 0) in CloudWatch Metric Math to visually tell a true gap apart from a legitimate zero in trend graphs.

Alerting on gaps

• Always present — absent() — Fires when the series disappears completely from the metric store
• Intermittent by design (batch, cron) — absent_over_time(metric[30m]) — Avoids false positives on short gaps

> [!IMPORTANT] A gap means no data was collected; a flat zero means the scrape worked and the value was zero. Diagnose them differently.

Key concepts

Staleness Marker

A special sentinel value Prometheus injects into a time series when a scrape target disappears, causing Grafana to render a gap rather than interpolating zero.

Scrape Failure

A failed HTTP request from Prometheus to a target's /metrics endpoint, recorded in the up metric as value 0.

absent()

PromQL function that returns 1 when a metric series does not exist in the current evaluation window — used to alert on missing series.

Clock Skew

A difference between the system clocks of the Prometheus server and a scrape target that causes samples to arrive out of order and be dropped.

FILL()

CloudWatch Metric Math function that replaces missing data points with a specified value (commonly 0) to distinguish gaps from zeros in visualizations.

When to use it

• Diagnosing broken lines or missing data points in Grafana dashboards
• Explaining why a Prometheus alert fired on absence of a metric series
• Investigating why CloudWatch dashboards show gaps after a deployment
• Designing monitoring systems resilient to pod restarts and label changes
• Debugging clock skew issues between scrape targets and Prometheus server

Interview questions

• How does a Prometheus staleness marker differ from a zero value, and how do you tell them apart in a dashboard?
• How do you alert when a critical metric series disappears entirely from Prometheus?
• What causes intermittent metrics gaps in a cross-AZ Prometheus setup?

More Troubleshooting topics

• IAM Permission Denied Errors
• JVM Garbage Collection Pauses
• Kubernetes Service Not Reachable
• Lambda Cold Start Latency
• Log Volume Explosion and Cost Runaway
• Memory Leak Diagnosis
• Missing Traces in Distributed Tracing
• Multi-Region High Latency
• NAT Gateway Bandwidth and Throttling
• Network Bandwidth Saturation
• OOMKilled Pod Diagnosis
• Packet Loss and Network Degradation