Overlay vs Underlay Networking

Underlay = real physical/cloud network that carries actual packets (routers, switches, data center, cloud VPC). Overlay = virtual network built on top using encapsulation/tunnels (VXLAN, Geneve, GRE). Kubernetes CNI uses overlay (VXLAN) to connect pods across nodes without changing physical network. Both work together: overlay needs healthy underlay.

Overview

Two networks, one packet

Every modern cloud and Kubernetes networking stack uses the overlay-underlay model. It explains how pods on different nodes communicate, why a packet trace shows two IP headers, and what happens when VXLAN breaks.

The underlay: the physical or cloud foundation

The underlay is the real network: physical routers, switches, cloud VPC routing tables, ISP links. It carries actual IP packets between machines and knows nothing about containers or Kubernetes. Its only job is to deliver packets from Node A to Node B based on real IP addresses.

AWS VPC — Routes between EC2 instances
Data center fabric — BGP between ToR switches
GCP VPC — Software-defined, but still the underlay from Kubernetes's perspective

> [!TIP] Analogy: the underlay is the highway system. It exists independently of the vehicles using it.

The overlay: a virtual layer on top

The overlay is a virtual network built on top of the underlay using encapsulation. Workloads get addresses in the overlay address space, which differ from the underlay addresses.

Example range — 10.244.0.0/16 for pods — 172.16.0.0/16 for nodes
Header in the packet — Inner: pod to pod — Outer: node to node

Common overlay technologies

VXLAN (Virtual Extensible LAN) — Most widely used; encapsulates Layer 2 frames in UDP; standard port 4789 — Flannel, Calico (overlay mode), Cilium (overlay mode), OpenShift SDN
Geneve — More extensible than VXLAN, same encapsulation concept — OVN-Kubernetes, NSX-T
GRE (Generic Routing Encapsulation) — Older, simpler, point-to-point tunnels — Less common in Kubernetes
IP-in-IP — Minimal overhead: IPv4 packets inside IPv4 — Calico in certain modes

Overlay or no overlay

VXLAN overlay — Default for Flannel and Calico; works on any network
Calico BGP mode — Programs BGP routes so routers reach each pod CIDR directly; no encapsulation overhead; faster, but needs underlay BGP support and is more complex to operate
Native VPC routing — Cloud providers (EKS, GKE, AKS) often use it for better performance
Cilium — Supports both overlay and native routing

The critical dependency: MTU

If the underlay has issues (packet loss, high latency, MTU mismatch), the overlay fails too. VXLAN adds about 50 bytes of header.

1500 — 1500 — 1550 — Fragmented or dropped
1500 — 1450 — 1500 — Fits

> [!IMPORTANT] Always configure the overlay MTU as the underlay MTU minus 50 bytes for VXLAN.

Key concepts

Underlay: The real physical or cloud network. Carries actual IP packets between machines. Knows only node/VM IP addresses. Examples: AWS VPC, data center routers/switches, ISP backbone. Stable, managed by infrastructure team or cloud provider.
Overlay: A virtual network built on top of the underlay using encapsulation. Workloads get overlay IP addresses (pod IPs). Packets between overlay addresses are wrapped in outer packets with underlay (node) addresses for transit. Managed by CNI plugins or SDN controllers.
VXLAN: Virtual Extensible LAN. Encapsulates Layer 2 frames (or Layer 3 packets, in practice) in UDP datagrams. Standard UDP port 4789. Adds approximately 50 bytes of overhead per packet. The dominant overlay technology for Kubernetes CNI plugins.
MTU (Maximum Transmission Unit): Maximum packet size a network link can carry. Standard Ethernet MTU: 1500 bytes. VXLAN adds ~50 bytes of encapsulation overhead. Pod MTU should be set to underlay MTU minus 50 (typically 1450) to prevent fragmentation. MTU mismatch causes mysterious packet drops for large payloads.
Encapsulation: Wrapping an inner packet (pod-to-pod payload) with an outer header (node-to-node routing header). The inner packet is the "cargo"; the outer header is the "shipping label." CNI agent on the source node adds the outer header; CNI agent on the destination node removes it.
BGP underlay mode (no overlay): Calico can advertise pod CIDRs as BGP routes to physical routers. Pods get routable IPs on the underlay — no encapsulation needed. Higher performance (no encap overhead, no MTU penalty) but requires BGP support from the underlay (not available on all cloud VPCs without additional config).

When to use it

Choosing a Kubernetes CNI plugin and understanding the performance implications
Debugging pod-to-pod connectivity across nodes — knowing that two IP headers exist in VXLAN traffic
Interview questions about cloud networking, SDN, or Kubernetes networking internals
Designing multi-cloud or hybrid cloud networking where overlay connects networks with incompatible addressing
Troubleshooting MTU-related packet drops in containerized environments

Interview questions

Explain the difference between overlay and underlay networking in Kubernetes.
How does VXLAN encapsulation work and what can go wrong?