TCP Congestion Control

Slow start, congestion avoidance, CUBIC and BBR algorithms, congestion window vs receive window, and tuning TCP for high-bandwidth long-latency paths.

Overview

Overview TCP congestion control prevents senders from overloading the network. Without it, a fast sender fills every buffer in the path, causing queue overflow and packet drops, which trigger retransmissions that worsen congestion (congestive collapse). TCP probes for available bandwidth while backing off when it detects congestion.

The sender maintains a congestion window (cwnd) limiting unacknowledged data in flight. Effective throughput = min(cwnd, rwnd) / RTT, where rwnd is the receiver's advertised window. Slow start doubles cwnd each RTT until reaching ssthresh or packet loss. Congestion avoidance grows cwnd linearly after that.

CUBIC (Linux default since 2.6.19) uses a cubic growth function for faster recovery on high-BDP paths. BBR (Bottleneck Bandwidth and Round-trip propagation time) takes a fundamentally different approach -- it measures the bottleneck bandwidth and minimum RTT to model the pipe, keeping exactly BDP worth of data in flight rather than reacting to packet loss. BBR dramatically improves throughput on lossy paths (satellite, 4G) where packet loss is not purely caused by congestion.

Key concepts

Bandwidth-Delay Product (BDP)
BDP = bandwidth * round-trip-time. It represents the data that can be in transit on the path at any instant to fully utilize the link. For a 1Gbps link with 100ms RTT: BDP = 1e9 * 0.1 = 100 megabits = 12.5MB. TCP needs at least 12.5MB in flight to saturate this link. If the socket buffer (SO_RCVBUF) is smaller than BDP, throughput is capped. Linux auto-tunes up to tcp_rmem_max (default ~6MB, often too small for cross-continent transfers).
BBR vs CUBIC
CUBIC reacts to packet loss: on loss, cut cwnd by 30% then grow using a cubic function. Better than Reno (50% cut, linear growth) on high-BDP paths. BBR does not use loss as a congestion signal. It continuously estimates bottleneck bandwidth (BtlBw) and minimum RTT (RTprop), then sets cwnd = BtlBw * RTprop. BBR periodically probes for more bandwidth by briefly increasing the send rate. On lossy paths (mobile, satellite), BBR maintains higher throughput because it does not interpret every lost packet as congestion.
Buffer Bloat
When network equipment has very large buffers, packet loss may not occur even under heavy congestion -- instead, RTT inflates as packets queue in the buffer. Loss-based algorithms keep increasing cwnd while RTT rises to hundreds of milliseconds. BBR and AQM (Active Queue Management: CoDel, fq_codel) detect and react to this earlier because they monitor RTT, not just loss.

When to use it

  • Diagnosing why throughput between datacenters is well below the link bandwidth (BDP not fully utilized).
  • Tuning Linux TCP parameters (tcp_rmem, tcp_wmem, tcp_congestion_control) for high-throughput transfers between AWS regions.
  • Explaining why a 100ms RTT link is harder to saturate than a 1ms RTT link at the same bandwidth.

Interview questions

  • You have a 10Gbps link between AWS regions with 80ms RTT but iperf3 shows only 200Mbps. What are the likely causes?

Sign up free to read the full topic

More Networking topics

Start free on Cariara · Coding problems · System design studio