TCP/IP Stack & Handshake

TCP 3-way handshake, SYN/ACK/FIN/RST flags, TCP state machine, congestion control algorithms, and Nagle algorithm tradeoffs.

Overview

Overview The TCP/IP stack is the foundational protocol suite that powers virtually all internet communication. Understanding it deeply is essential for any engineer working on distributed systems, backend services, or infrastructure.

TCP (Transmission Control Protocol) operates at Layer 4 of the OSI model and provides reliable, ordered, and error-checked delivery of data between applications. It achieves this through a combination of sequence numbers, acknowledgments, retransmissions, and flow control mechanisms.

The 3-way handshake is the mechanism by which a TCP connection is established. The client sends a SYN (synchronize) segment, the server responds with SYN-ACK (synchronize-acknowledge), and the client completes the handshake with an ACK (acknowledge). This exchange allows both sides to agree on initial sequence numbers and establish connection parameters. Tearing down a connection is a 4-way process: FIN, ACK, FIN, ACK — because each direction of data flow is closed independently.

TCP flags carry semantic meaning: SYN initiates connections, ACK acknowledges received data, FIN initiates graceful close, RST abruptly terminates a connection, PSH requests immediate delivery to the application layer, and URG marks urgent data.

Congestion control is one of TCP's most sophisticated features. Algorithms like CUBIC (default on Linux), BBR (Bottleneck Bandwidth and Round-trip propagation time), and Reno use different signals — packet loss, RTT variance, and explicit congestion notification (ECN) — to probe available bandwidth without overwhelming the network.

The Nagle algorithm reduces small-packet overhead by coalescing writes into fewer, larger segments. While this improves throughput on high-latency links, it introduces latency for interactive applications like SSH and database protocols, which often disable it with TCP_NODELAY.

In cloud and SRE contexts, understanding TCP's state machine, TIME_WAIT accumulation, SYN flood protection, and keep-alive settings is critical for diagnosing connection exhaustion, latency spikes, and cascading failures under load.

Key concepts

3-Way Handshake
The TCP connection establishment sequence: client sends SYN with its initial sequence number (ISN), server responds with SYN-ACK containing its own ISN and acknowledging the client's ISN+1, then the client sends ACK. After this, the connection enters ESTABLISHED state on both sides.
TCP State Machine
TCP connections transition through a defined set of states: CLOSED, LISTEN, SYN_SENT, SYN_RECEIVED, ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, CLOSE_WAIT, CLOSING, LAST_ACK, TIME_WAIT. TIME_WAIT lasts 2*MSL (typically 60-120 seconds) to ensure delayed packets from the old connection do not corrupt a new one on the same 4-tuple.
Congestion Control
TCP's mechanism to avoid overwhelming the network. It uses a congestion window (cwnd) that starts small and grows. Slow Start doubles cwnd each RTT until a threshold is reached, then Congestion Avoidance increases linearly. On loss, the window is reduced. Modern algorithms like BBR model the bottleneck bandwidth rather than relying solely on loss signals.
Nagle Algorithm
An optimization that buffers small TCP writes and sends them as one segment when either the previous send is acknowledged or the buffer reaches MSS. Reduces chatty small-packet traffic but adds up to one RTT of latency per write. Disabled with the TCP_NODELAY socket option for latency-sensitive protocols.
RST vs FIN
FIN initiates a graceful half-close: data already in flight is delivered before the connection closes. RST abruptly terminates the connection; any unread data is discarded. RST is sent when a packet arrives for a non-existent connection, when a process crashes without closing sockets, or when a firewall drops a connection mid-stream.
TCP Keep-Alive
An optional mechanism where the kernel sends empty ACK probes after a period of inactivity to detect dead peers. Controlled by tcp_keepalive_time, tcp_keepalive_intvl, and tcp_keepalive_probes. Not enabled by default on sockets; must be set with SO_KEEPALIVE. Application-level heartbeats are often preferred for faster detection.

When to use it

  • Diagnosing connection establishment failures or latency spikes between microservices
  • Tuning TCP socket options for high-throughput or low-latency services
  • Investigating SYN floods, TIME_WAIT accumulation, or connection pool exhaustion
  • Understanding why a service is slow to recover after a network partition
  • Evaluating whether to use TCP keep-alives or application-level heartbeats

Interview questions

  • Walk me through the TCP 3-way handshake and explain what information is exchanged at each step.
  • Explain TCP congestion control. How does BBR differ from CUBIC, and when would you choose one over the other?
  • What happens to TCP connections when a server crashes mid-connection, and how do you detect and handle this in production?

Sign up free to read the full topic

More Networking topics

Start free on Cariara · Coding problems · System design studio