Packet Loss and Network Degradation

Distinguish packet loss from latency using retransmission metrics and MTU diagnostics.

Overview

Overview

Packet loss is deceptive because TCP hides it: applications rarely see errors, only lower throughput and higher latency while TCP retransmits and shrinks its congestion window. A 1% loss rate can cut TCP throughput by up to 50% on high-bandwidth connections, so the app works but is slow with nothing obvious to trace.

Loss vs latency

  • Transfer pattern — Normal throughput interrupted by pauses while TCP times out and retransmits — Uniformly slow
  • Retransmission counters — Increasing — Flat

Commands and what they tell you

  • netstat -s — TcpExt:TCPRetransFail and TcpExt:TCPTimeouts; rising values mean retransmissions
  • ss -tin — Retransmission counters per connection, in real time
  • ethtool -i eth0 — Driver name; ena means enhanced networking is active
  • VPC Flow Logs filtered on REJECT — Connections dropped by security groups or NACLs before they reach the app

> [!NOTE] Flow Logs do not capture application-level issues, but they are definitive for network-level drops.

MTU mismatch

EC2 instances inside a VPC support 9001-byte jumbo frames, while internet, VPN and Direct Connect paths typically use 1500 bytes. Path MTU Discovery relies on ICMP "fragmentation needed" messages, and many firewalls and security groups block ICMP.

> [!WARNING] The signature: the app works for small payloads and fails or hangs for large ones.

Stateful SGs vs stateless NACLs

Security group and NACL misconfiguration is the most common cause of asymmetric packet loss in AWS.

  • Security group — Stateful; allowed automatically for an allowed inbound connection — Low
  • NACL — Stateless; inbound and outbound must both be allowed — Inbound 443 without outbound 1024-65535 drops all HTTPS return traffic

> [!IMPORTANT] A NACL return-path drop is invisible in the security group view but kills connections.

Enhanced networking

The ENA driver supports up to 100 Gbps but must be enabled on the instance type and present as a driver. Without ENA on instances that support it, you cap at 1 Gbps even on types rated higher.

Key concepts

TCP Retransmission
TCP's mechanism for recovering lost packets — if an acknowledgment is not received within the retransmit timeout (RTO), TCP resends the segment and halves the congestion window, dramatically reducing throughput.
Path MTU Discovery (PMTUD)
The process by which TCP determines the maximum packet size the network path supports. Relies on ICMP "fragmentation needed" messages — blocking ICMP causes PMTUD failures and silent packet drops for large payloads.
Stateful vs Stateless Firewall
Security groups track connection state and automatically allow return traffic. NACLs are stateless and require explicit allow rules for both directions — asymmetric NACL rules are a common source of mysterious packet drops.
Jumbo Frames
Ethernet frames with MTU up to 9001 bytes, supported within EC2 VPCs. Using jumbo frames between instances within a VPC improves throughput for large transfers but requires MTU to drop to 1500 for external connections.
ENA (Elastic Network Adapter)
The enhanced networking driver for EC2 that enables high-bandwidth networking up to 100 Gbps. Required to achieve the full bandwidth of instance types rated above 10 Gbps.

When to use it

  • Diagnosing applications that are slow without obvious errors, suggesting TCP retransmissions
  • Explaining the difference between security group and NACL behavior in networking interviews
  • Investigating MTU mismatches causing failures for large payloads only

Interview questions

  • A service running on EC2 can reach an external API for small requests but hangs or times out when sending payloads larger than about 1400 bytes. Small GET requests work fine. What is the likely cause and how do you verify?

Sign up free to read the full topic

More Troubleshooting topics

Start free on Cariara · Coding problems · System design studio