Network Chaos Engineering

Overview

Overview Network chaos engineering deliberately injects network failures — latency, packet loss, bandwidth limits, corruption, reordering — to verify that distributed systems behave correctly under adverse conditions.

Linux tc netem (network emulator): tc qdisc add dev eth0 root netem delay 100ms 20ms distribution normal tc qdisc add dev eth0 root netem loss 1% 25% # correlated loss tc qdisc add dev eth0 root netem corrupt 0.1% tc qdisc add dev eth0 root netem duplicate 1% reorder 5% 50%

Toxiproxy (Shopify): a TCP proxy that injects network conditions at the application layer. Supports: latency, slow_close, timeout, bandwidth, slicer (chunked delivery), reset_peer. Configurable via HTTP API — ideal for integration tests. Run as a sidecar in CI/CD.

Chaos Mesh (CNCF): Kubernetes-native chaos engineering. NetworkChaos CRD supports: network partition, delay, loss, corrupt, bandwidth, duplicate. Applies faults using tc and iptables under the hood via a privileged DaemonSet. Faults can be targeted to specific pod label selectors and namespaces.

What to test: - Timeout behavior: does the service return an error within its declared timeout? Does it release resources? - Retry storms: under 50% packet loss, do retries amplify load on downstream services? - Circuit breaker activation: does the circuit open under sustained failure rates? - Graceful degradation: does the service fall back to cached data under 1000ms backend latency? - Cascading failure prevention: does a single slow service degrade the entire request path?

Key concepts

tc netem

Linux traffic control network emulator. Injects delay, loss, duplication, reordering, and corruption on a network interface. Requires root or CAP_NET_ADMIN. Applied via tc qdisc commands.

Toxiproxy

Shopify's TCP proxy for chaos testing. Injects network conditions via HTTP API. Runs in CI/CD to test timeout and retry logic without kernel-level privileges.

Chaos Mesh NetworkChaos

Kubernetes CRD for network fault injection. Targets specific pods by label selector. Implements faults via tc and iptables on node DaemonSets. Supports partition, delay, loss, corrupt, bandwidth.

Retry storm

When a downstream service slows, upstream services retry — amplifying load on the already-slow service. Exponential backoff with jitter prevents synchronized retry waves.

Circuit breaker

Pattern that stops sending requests to a failing service after a threshold of errors. Returns a fast error immediately instead of waiting for timeout. Prevents cascading failures. Implemented in Istio via DestinationRule outlier detection.

When to use it

• Verifying circuit breaker configuration before a production deployment
• Testing that timeouts are set correctly across all service dependencies
• Validating graceful degradation under database latency spikes
• CI/CD integration tests that include network fault scenarios

Interview questions

• How do you use tc netem to simulate a flaky network for integration testing?

More Networking topics

• Latency Diagnosis
• Load Balancing Algorithms
• MPLS
• mTLS & SPIFFE/SPIRE Workload Identity
• MTU & IP Fragmentation
• NAT & PAT
• Network Debugging Toolkit
• Network Observability & Flow Analysis
• nginx and HAProxy Configuration
• OSI & TCP/IP Models
• OSPF & EIGRP
• Overlay vs Underlay Networking