IPSec & VPN Deep Dive

Overview

Overview IPSec (Internet Protocol Security) is a suite of protocols for authenticating and encrypting IP traffic. It operates at Layer 3, making it transport-agnostic — any TCP/UDP/ICMP traffic is protected transparently.

IPSec modes: - Transport mode: encrypts only the IP payload (TCP/UDP segment). Original IP header is preserved. Used for host-to-host or host-to-gateway communication. - Tunnel mode: encrypts the entire original IP packet and wraps it in a new IP header. Used for site-to-site VPNs — the original packet is hidden inside the tunnel. The new outer IP header routes between VPN gateways.

IPSec protocols: - ESP (Encapsulating Security Payload): provides confidentiality (encryption), integrity, and optional authentication. The standard choice for VPNs. - AH (Authentication Header): provides integrity and authentication but no encryption. Rarely used alone; incompatible with NAT (AH covers the outer IP header, which NAT modifies).

IKE (Internet Key Exchange): negotiates and manages IPSec SAs (Security Associations). - IKEv1: two phases. Phase 1: establishes IKE SA (authenticated channel). Phase 2: negotiates IPSec SA (encryption parameters for data). Main mode (6 messages) or aggressive mode (3 messages, less secure). - IKEv2: single exchange, faster, more reliable NAT traversal, built-in EAP support, dead peer detection. Preferred for all new deployments.

AWS Site-to-Site VPN: managed IPSec VPN. Creates two tunnels (active/standby) between a Virtual Private Gateway (VGW) or Transit Gateway and the customer gateway (on-premises device). Uses IKEv2 by default (IKEv1 available). Pre-shared keys or certificates. BGP optional for dynamic routing.

Key concepts

Security Association (SA)

A one-way logical connection with defined encryption/authentication parameters. Full IPSec communication requires two SAs (one per direction). SAs are identified by SPI (Security Parameter Index).

IKE Phase 1

Establishes an authenticated, encrypted IKE SA between peers. Negotiates: encryption algorithm, hash, DH group, and authentication method (PSK or certificates). Result: ISAKMP SA.

IKE Phase 2

Uses the IKE SA to negotiate IPSec SAs for data. Negotiates: ESP algorithm, hash, PFS (perfect forward secrecy), and selectors (which traffic to encrypt). Result: IPSec SA pair.

NAT-T (NAT Traversal)

Encapsulates ESP in UDP port 4500 to traverse NAT devices. Required when either endpoint is behind NAT, since NAT modifies IP headers that ESP protects.

PFS (Perfect Forward Secrecy)

Each Phase 2 SA uses a new Diffie-Hellman key exchange. Compromising the long-term private key cannot decrypt past sessions. IKEv2 with PFS is the recommended configuration.

When to use it

• Designing hybrid cloud connectivity between on-premises data center and AWS/GCP
• Debugging AWS Site-to-Site VPN phase 1 or phase 2 negotiation failures
• Explaining why AH is incompatible with NAT
• Choosing between IPSec, WireGuard, and OpenVPN for a new VPN deployment

Interview questions

• What is the difference between IPSec transport mode and tunnel mode?
• How do you debug a failing AWS Site-to-Site VPN tunnel?

More Networking topics

• gRPC vs REST
• Health Checks and Circuit Breaking
• HTTP/2 & HTTP/3
• HTTP/HTTPS & TLS
• Hybrid Cloud Connectivity
• IP Addressing & CIDR
• iptables and nftables
• IPv6 Addressing & Dual Stack
• Kubernetes Network Policies
• L4 vs L7 Load Balancing
• Latency Diagnosis
• Load Balancing Algorithms