Network Debugging Toolkit

Layered debugging methodology, tcpdump, Wireshark, mtr, netstat, and cloud-specific tools.

Overview

Effective network debugging follows a structured layered approach: start at the lowest layer that could explain the symptom and work upward. Form a hypothesis about which layer is failing, then choose the tool that confirms or refutes it.

Debugging ladder by layer

Layer 1-2 (Physical/Link): ip -s link show — check error counters, drops, collisions. ethtool eth0 — link speed, duplex, PHY errors.
Layer 3 (IP): ping (ICMP reachability), ip route get (routing decision), traceroute / mtr (path and per-hop latency).
Layer 4 (TCP): nc -zv host port (can we connect?), ss -tulpn (what is listening?), tcpdump (what is on the wire?).
Layer 7 (Application): curl -v (HTTP), openssl s_client (TLS), dig (DNS).

Key tools

mtr combines traceroute and ping: continuously probes each hop and shows real-time loss% and latency. Essential for finding which hop is dropping packets.

ss (socket statistics) is the production standard. Common invocations: - ss -s — summary of connection counts - ss -tulpn — all listening sockets with process names and PIDs - ss -t state established — all established TCP connections - ss -t state time-wait — TIME_WAIT count (useful for diagnosing port exhaustion)

Cloud-specific tools

VPC Flow Logs — per-ENI accept/reject records; invaluable for diagnosing security group/NACL issues
Route53 Resolver Query Logs — DNS queries from VPC resources
CloudFront access logs — edge cache behavior and error codes
CloudWatch Network Monitor — continuous network latency and packet loss metrics

Key concepts

mtr: Combines traceroute and ping. Continuously probes each hop and shows real-time loss% and latency. Essential for finding which specific hop is dropping or delaying packets.
VPC Flow Logs: AWS per-ENI network logs: source IP, destination IP, port, protocol, bytes, and ACCEPT/REJECT action. Stored in CloudWatch Logs or S3. Query with Logs Insights to diagnose SG/NACL issues.
ss -tulpn: List all listening TCP/UDP sockets with process name and PID. Faster than netstat. Use to verify a service is listening on the expected port and interface (0.0.0.0 vs 127.0.0.1).
tcpdump filtering: BPF filter syntax: host 10.0.1.5 and port 5432 and tcp — combines IP, port, and protocol filters. Use not port 22 to exclude SSH noise.
MTU issues: Maximum Transmission Unit. AWS instances use MTU 9001 (jumbo frames) within a VPC. Crossing an IGW or VPN drops to 1500. PMTUD (Path MTU Discovery) uses ICMP type 3 code 4 — blocking ICMP causes silent packet drops.

When to use it

Systematically diagnosing a network connectivity issue in production
Using mtr to find which network hop is causing packet loss
Reading VPC Flow Logs to determine if a security group is rejecting traffic
Building a network debugging runbook for an on-call team

Interview questions

A microservice in Kubernetes cannot reach another microservice in a different namespace. Walk through your diagnosis.