Network Bandwidth Saturation

Detect and resolve network bandwidth saturation on EC2 instances and container workloads.

Overview

Overview

Network bandwidth saturation happens when a host, instance or network path hits its maximum transfer rate, causing packet drops, higher latency and connection timeouts. CPU and memory are closely watched, but bandwidth limits often stay invisible until they cause failures, because many setups never compare absolute bandwidth against the instance's limit.

EC2 bandwidth limits

Each instance type and size has a baseline and a burst limit. Exceeding the baseline on a burstable instance drains a network credit bucket; sustained traffic that empties it is throttled to baseline.

  • t3.medium — Baseline 0.256 Gbps, burst to 5 Gbps
  • c5.xlarge, c5.4xlarge — Up to 10 Gbps (burst; the baseline is lower)
  • c5.9xlarge — 12 Gbps
  • c5.18xlarge — 25 Gbps
  • c5n, m5n ("n" = network-optimized) — 25 to 100 Gbps, built for network-heavy workloads

> [!NOTE] Corrected: "up to 10 Gbps" sizes such as c5.xlarge burst to 10 Gbps from a lower baseline (about 1.25 Gbps for c5.xlarge), and t3.medium's baseline is 0.256 Gbps.

Commands and what they tell you

  • CloudWatch NetworkIn / NetworkOut — Bytes per period; divide by the period length for bytes per second and compare with the instance limit
  • sar -n DEV 1 — Per-second bytes and packets per interface; run for 60 seconds during a spike
  • iftop (usually with sudo) — Live bandwidth by connection pair; shows at once if one destination dominates
  • nethogs — Bandwidth grouped by process: one application or many
  • CNI plugins (Weave, Cilium and similar) — Pod-level traffic; else kubectl top pods if available, or node-level iftop filtered by pod IP range
  • VPC Flow Logs + Athena or CloudWatch Insights — Retrospective top-N IP pairs by bytes

A flow-log query for the top pairs: SELECT srcaddr, dstaddr, SUM(bytes) as total_bytes FROM flow_logs GROUP BY srcaddr, dstaddr ORDER BY total_bytes DESC LIMIT 20. It surfaces cross-region copies, log shipping to unexpected destinations and traffic amplification from large responses.

Fixes by cause

  • Legitimate traffic outgrew the instance — Network-optimized instance type, or enable enhanced networking (ENA)
  • Inefficient transfer — HTTP gzip for API responses: 60-80% less bandwidth for JSON
  • Moving data to the processing — Process in the same AZ as the data; filter server-side instead of downloading whole objects
  • Large object uploads — Multipart uploads parallelize without raising per-connection bandwidth

> [!TIP] Compression is usually the cheapest first fix for API-heavy services.

Key concepts

EC2 Network Baseline
The guaranteed minimum network bandwidth for an EC2 instance type — smaller instances can burst above this but are throttled back to baseline when network credits are exhausted.
Enhanced Networking
An EC2 feature (enabled via the ENA adapter) that provides higher packet-per-second performance and lower latency for supported instance types.
VPC Flow Logs
Per-ENI records of all IP traffic including source/destination IP, port, protocol, and byte count — used for security analysis and identifying top bandwidth consumers.
iftop
A real-time network bandwidth monitor that shows data transfer rates by connection pair — useful for quickly identifying which remote hosts are consuming the most bandwidth.
Bandwidth Amplification
When a small request triggers a disproportionately large response — for example, a query that returns megabytes of JSON where kilobytes would suffice — a common hidden cause of bandwidth saturation.

When to use it

  • Diagnosing intermittent connection timeouts that correlate with high network traffic
  • Identifying which process or service is consuming disproportionate network bandwidth
  • Explaining EC2 network baseline vs burst limits and when they cause throttling
  • Using VPC Flow Logs to find unexpected data transfers between services
  • Designing a content compression and data locality strategy to reduce bandwidth consumption

Interview questions

  • How do you identify the root cause of network bandwidth saturation on an EC2 instance?
  • How do VPC Flow Logs help diagnose bandwidth saturation and unexpected data transfers?
  • What compression and data locality strategies most effectively reduce network bandwidth consumption?

Sign up free to read the full topic

More Troubleshooting topics

Start free on Cariara · Coding problems · System design studio