Overlay vs Underlay Networking

Underlay is the real physical network. Overlay is a virtual network built on top using encapsulation (VXLAN, GRE, Geneve). VXLAN most widely used in Kubernetes CNI, SDN, and cloud.

Overview

Underlay Network The underlay is the real physical network infrastructure — the actual switches, routers, cables, and IP addressing that physically exists. It provides basic IP connectivity between nodes. The underlay only needs to route IP packets between endpoints; it has no knowledge of the virtual networks running on top.

Overlay Network An overlay is a virtual network built on top of the underlay using encapsulation. Overlay protocols wrap (encapsulate) original network frames inside another protocol's packets, creating logical tunnels across the physical network.

Encapsulation Protocols

VXLAN — L2 frame in UDP — 4789/UDP — Kubernetes CNI, SDN, cloud
GRE — IP in GRE header — Protocol 47 — Point-to-point tunnels
Geneve — L2 frame in UDP — 6081/UDP — Extensible, SDN controllers
IP-in-IP — IP packet in IP — Protocol 4 — Simple tunneling

VXLAN — Most Widely Used VXLAN (Virtual Extensible LAN) is the dominant overlay protocol: - Encapsulates L2 Ethernet frames inside UDP packets - Adds 50-byte overhead (14 VXLAN + 8 UDP + 20 IP + 14 Ethernet outer) - Default UDP port 4789 - Uses 24-bit VNI (VXLAN Network Identifier) — supports 16 million virtual networks vs VLAN's 4096

Kubernetes Use Kubernetes CNI plugins use overlay networking to connect pods across nodes: - Flannel: simple VXLAN overlay, all pods in one flat network - Calico: can use VXLAN or IP-in-IP overlay, or pure BGP underlay routing - Weave: VXLAN or encrypted overlay - Cilium: eBPF-based, supports VXLAN overlay or native routing

Key concepts

Underlay: The physical IP network providing basic connectivity between nodes. Routers and switches form the underlay. It only needs to forward IP packets between endpoints — no knowledge of virtual networks.
Overlay: A virtual network built on top of the underlay using encapsulation. Creates logical tunnels that appear as direct connections between endpoints even when they are physically distant.
VXLAN: Virtual Extensible LAN. Encapsulates L2 Ethernet frames in UDP (port 4789). Adds 50-byte overhead. Supports 16M virtual networks via 24-bit VNI. Most common overlay in Kubernetes and cloud.
VTEP (VXLAN Tunnel Endpoint): The endpoint that performs VXLAN encapsulation and decapsulation. In Kubernetes, each node runs a VTEP. The VTEP encapsulates outgoing frames and decapsulates incoming VXLAN packets.
VNI (VXLAN Network Identifier): 24-bit field in the VXLAN header identifying the virtual network segment. Allows 16,777,216 distinct virtual networks on the same underlay — vs VLAN's 12-bit 4096 limit.
MTU considerations: VXLAN adds 50 bytes of overhead. If the underlay MTU is 1500, the overlay effective MTU is 1450. Mismatched MTU causes silent packet drops for large frames. Always set underlay MTU to 1550+ or configure overlay MTU to 1450.

When to use it

Designing Kubernetes pod networking — choosing between VXLAN overlay (simpler) vs BGP underlay (better performance)
Building multi-tenant cloud networks where different customers need isolated L2 segments across shared physical infrastructure
Explaining how AWS VPC, Azure VNet, and GCP VPC isolate tenant traffic on shared hardware (all use VXLAN-like encapsulation)
Troubleshooting MTU issues — VXLAN's 50-byte overhead requires MTU 1550 on the underlay if VMs use MTU 1500

Interview questions

What is the difference between overlay and underlay networking?
How does VXLAN work and why is it used in Kubernetes?
What are the tradeoffs between VXLAN overlay and BGP-based underlay routing for Kubernetes?