Diagnosing Connectivity Failures

Interpreting ping and traceroute output, firewall drop vs reject behavior, MTU black holes, and asymmetric routing issues.

Overview

Overview Connectivity failures are one of the most common and frustrating categories of production incidents. A service that worked yesterday suddenly cannot be reached, or a new deployment fails to connect to a dependency. Systematic diagnosis using the right tools and a layered OSI-model approach dramatically reduces mean time to resolution.

The fundamental diagnostic loop is: determine what layer the failure is at (physical, IP, transport, application), then use layer-appropriate tools to isolate the exact failure point. Starting with ping and traceroute provides an initial map of IP-layer connectivity. If IP connectivity exists but TCP connections fail, tcpdump at both endpoints reveals whether SYN packets are arriving and whether RST or ICMP unreachable messages are being sent back. If TCP connects but application responses are wrong, curl and application-layer debugging tools take over.

Understanding the difference between ICMP unreachable (connection rejected, usually a firewall rule returning an explicit deny) and connection timeout (packet dropped silently, usually a stateful firewall or network ACL with no return) is critical. A firewall that sends TCP RST or ICMP port-unreachable is actually more helpful for diagnosis than one that silently drops — the error is immediate and informative. A silent drop causes the client to wait for the full connection timeout (typically 120 seconds) before failing.

MTU black holes are a notoriously difficult class of connectivity failure. TCP connections establish successfully (SYN/SYN-ACK/ACK use small packets) but data transfer fails or hangs. This happens when the Path MTU is smaller than the sender assumes, the DF (Don't Fragment) bit is set, and ICMP "fragmentation needed" messages are blocked by a firewall. The result is silent packet drops for any packet larger than the Path MTU.

In cloud environments, additional failure modes include security group misconfiguration, network ACL issues, route table problems, and DNS resolution failures that masquerade as connectivity issues.

Key concepts

ICMP Unreachable vs Silent Drop
A firewall returning ICMP port-unreachable or TCP RST tells the client immediately that the connection is refused. The client gets an immediate error (Connection refused). A silent drop causes the client to wait until the connection attempt timeout expires (TCP SYN is retransmitted 3-6 times over 60-120 seconds before giving up).
Traceroute Interpretation
Traceroute sends packets with incrementing TTL values. Each hop decrements TTL by 1; when TTL reaches 0 the router sends ICMP Time Exceeded back, revealing its IP. Asterisks indicate the router did not respond to the probe — not necessarily a failure; many routers rate-limit or drop ICMP TTL-exceeded. A final asterisk means the destination did not respond.
Asymmetric Routing
When forward-path packets traverse different routers than return-path packets. Stateful firewalls track connection state: if they see a TCP SYN on the forward path but the SYN-ACK returns via a different path (not through the firewall), the firewall has no state for the flow and drops the SYN-ACK. Results in connections that initiate but never complete.
MTU Black Hole
Occurs when Path MTU Discovery fails because ICMP Fragmentation Needed messages (type 3, code 4) are blocked by firewalls. The sender sets DF bit, sends a packet larger than a link in the path can handle, the intermediate router cannot fragment (DF set), sends ICMP Frag-Needed, but that ICMP is blocked. The sender never learns to use a smaller MTU and retransmits silently discarded large packets.
Network ACL vs Security Group
In AWS, Security Groups are stateful (return traffic automatically allowed), applied at the ENI level, and support allow-only rules. Network ACLs are stateless (must explicitly allow both inbound and outbound), applied at the subnet level, support both allow and deny rules, and process rules in numbered order (lowest first). A common mistake is configuring the SG correctly but forgetting to allow the return traffic in the NACL.

When to use it

  • Responding to service unavailability incidents where the root cause is network-related
  • Validating network configuration after infrastructure changes or deployments
  • Diagnosing intermittent connectivity issues that do not appear in application logs
  • Investigating asymmetric routing causing stateful firewall session drops
  • Troubleshooting cross-region or cross-account connectivity in AWS environments

Interview questions

  • Walk through your systematic process for diagnosing "service X cannot reach service Y" in a Kubernetes cluster deployed on AWS.
  • How do you diagnose and fix an MTU black hole causing TCP connections to hang after the handshake?

Sign up free to read the full topic

More Networking topics

Start free on Cariara · Coding problems · System design studio