Linux Networking L2 and L3

Overview

Overview Linux networking is built on a layered model where Layer 2 and Layer 3 each have distinct responsibilities but cooperate tightly inside the kernel. Layer 2, the data link layer, concerns itself with how bytes travel between two directly connected devices: it defines the Ethernet frame format (destination MAC, source MAC, EtherType, payload, FCS), address resolution via ARP, and logical grouping of interfaces using Linux bridges and 802.1Q VLAN tagging. Layer 3, the network layer, introduces the concept of logical addressing with IP, autonomous routing decisions based on destination prefix lookups, and policy-based routing that can forward packets based on source IP, firewall mark, or DSCP bits.

The netfilter framework is the kernel subsystem that ties these layers together with stateful packet inspection and manipulation. Every packet that enters, traverses, or leaves a Linux machine visits a sequence of netfilter hooks. Tables contain chains of rules, and iptables is the classical userspace tool for managing those rules. nftables is the modern replacement with a unified syntax, but iptables remains dominant in production environments, container runtimes, and Kubernetes data planes, so understanding both is essential for any DevOps or infrastructure engineer.

Linux network namespaces virtualize the network stack itself. A namespace owns a private view of interfaces, routing tables, iptables rules, ARP caches, and sockets. This is the kernel primitive that makes containers possible: each container runs inside its own namespace, isolated from every other container and from the host. Veth pairs are the wire between namespaces: they behave like a crossed Ethernet cable at the kernel level, delivering frames from one end to the other with essentially zero latency and no intermediate switching logic.

Docker and Kubernetes build their entire networking models on top of these primitives. Docker creates a Linux bridge (docker0), provisions veth pairs for each container, and installs iptables MASQUERADE rules so outbound traffic can reach the internet. Kubernetes delegates network setup entirely to CNI plugins, which are short-lived binaries invoked by the container runtime. The CNI specification defines a precise contract: the runtime passes configuration on stdin and reads a JSON result from stdout, allowing any conforming plugin to wire up pod networking without modifying the runtime.

Understanding this stack end-to-end is required for debugging packet drops (ip route, ip neigh, conntrack, iptables -L -n -v), for designing multi-tenant network isolation (namespaces, VLAN tags, policy routing tables), and for reasoning about the performance characteristics and failure modes of container networking overlays. A candidate who can trace a packet from a containerized application through every kernel layer to the physical wire, and back, demonstrates the depth needed for senior infrastructure, SRE, and platform engineering roles.

Key concepts

Ethernet Frame and ARP

An Ethernet frame is the fundamental unit of Layer 2 communication. Its structure is: 6-byte destination MAC, 6-byte source MAC, optional 4-byte 802.1Q VLAN tag (EtherType 0x8100 followed by PCP, DEI, and 12-bit VID), 2-byte EtherType (0x0800 for IPv4, 0x0806 for ARP, 0x86DD for IPv6), variable payload (46-1500 bytes for standard MTU), and 4-byte FCS. When a host needs to send an IP packet to a destination on the same subnet, it must resolve the destination IP to a MAC address. The Address Resolution Protocol handles this: the sender broadcasts an ARP request containing the target IP, and the owner of that IP replies with its MAC. Linux caches these mappings in the neighbor (ARP) table managed by the kernel's neighbour subsystem. ARP operates only within a broadcast domain. Routers do not forward ARP requests, so each subnet requires its own ARP resolution. In Kubernetes, each node is its own broadcast domain at Layer 2; pod-to-pod communication across nodes therefore relies on Layer 3 routing (or encapsulation) rather than ARP.

Linux Bridge and 802.1Q VLAN Tagging

A Linux bridge is a software implementation of an Ethernet switch. It maintains a forwarding database (FDB) mapping MAC addresses to bridge ports. When a frame arrives on a port, the bridge looks up the destination MAC: if found, the frame is forwarded to the specific port; if not, it is flooded to all ports except the one it arrived on. The bridge learns source MACs from incoming frames and ages them out. Docker enslaves each container's host-side veth to the docker0 bridge, so containers on the same host communicate at Layer 2 without leaving the kernel. 802.1Q VLAN tagging inserts a 4-byte tag into the Ethernet frame between the source MAC and EtherType fields. The 12-bit VID field allows 4094 distinct VLANs. Linux VLAN subinterfaces strip the tag on ingress and insert it on egress, presenting a clean untagged interface to upper-layer protocols. Bridge VLAN filtering (bridge link set dev eth1 pvid 100 vid 100) can enforce VLAN membership at the port level, providing isolation equivalent to a managed switch.

IP Routing Table and ECMP

The Linux routing table maps destination IP prefixes to nexthops. The kernel performs longest-prefix-match (LPM) lookup for every outbound packet. A route entry specifies the destination network, the nexthop (gateway IP or directly connected), the output interface, and metric. The main routing table (table 254) handles normal traffic; local (table 255) handles loopback and broadcast; policy routing adds user-defined tables (1-252) selected by ip rules. ECMP (Equal-Cost Multi-Path) allows the kernel to distribute flows across multiple nexthops. Linux uses a hash of the 5-tuple (src IP, dst IP, protocol, src port, dst port) to select the nexthop, ensuring that all packets of a single TCP connection take the same path (preventing reordering). The sysctl net.ipv4.fib_multipath_hash_policy controls the hash inputs.

Netfilter Tables and Chains

Netfilter is the kernel framework that processes packets at five hook points. iptables organizes rules into tables, each with a fixed set of chains. The raw table (chains: PREROUTING, OUTPUT) runs before connection tracking and is used to exempt specific flows via the NOTRACK target. The mangle table (all five chains) allows modification of packet fields like TTL, ToS, and firewall mark. The nat table (PREROUTING, INPUT, OUTPUT, POSTROUTING) performs address translation; DNAT rules appear in PREROUTING and SNAT/MASQUERADE appear in POSTROUTING. The filter table (INPUT, FORWARD, OUTPUT) is the default for allow/deny rules. Connection tracking (conntrack) is the kernel module that maintains state for each flow. States are NEW (first packet of a new connection), ESTABLISHED (bidirectional traffic seen), RELATED (new connection related to an existing one, such as FTP data), and INVALID (packets that do not match any connection). Most FORWARD chains use -m state --state RELATED,ESTABLISHED to pass return traffic without explicit rules.

Linux Network Namespaces and Veth Pairs

A network namespace is a complete virtualization of the Linux network stack: it owns its own interfaces, routing tables, iptables rules, neighbor cache, and socket namespace. The root namespace (pid 1's namespace) is the default. New namespaces are created with ip netns add or unshare --net. Processes join a namespace by calling setns() on the namespace file descriptor. A veth pair is created as a single unit: two virtual Ethernet interfaces that are connected back-to-back in the kernel. A frame written to one end is immediately readable on the other end. They are the standard mechanism for connecting a network namespace to the root namespace or to a bridge. Named network namespaces appear as files under /var/run/netns/. Container runtimes create anonymous namespaces via clone() or unshare() and reference them through /proc/PID/ns/net. The CNI specification requires the runtime to pass this path in the CNI_NETNS environment variable so the CNI plugin can open and configure the namespace.

CNI Plugin Contract

The Container Network Interface specification defines a binary interface between a container runtime and a network plugin. At pod creation, the runtime executes a CNI binary with these environment variables set: CNI_COMMAND (ADD, DEL, or CHECK), CNI_CONTAINERID (opaque unique ID), CNI_NETNS (path to the network namespace file, e.g. /proc/12345/ns/net), CNI_IFNAME (interface name inside the namespace, typically eth0), CNI_ARGS (optional semicolon-separated key=value pairs), and CNI_PATH (colon-separated list of directories to search for plugin binaries). The plugin reads a JSON network configuration from stdin. A typical bridge plugin config looks like this: On ADD the plugin must: create the bridge if absent, create a veth pair, move one end into the container namespace and rename it to CNI_IFNAME, enslave the host-side veth to the bridge, invoke the IPAM delegate to allocate an IP, assign the IP to the container interface, add routes inside the namespace, and write a JSON result to stdout containing the allocated IPs and routes. On DEL the plugin must tear down the veth pair and release the IP back to IPAM. On CHECK it must verify the configuration matches expectations and return an error if not. The host-local IPAM plugin stores allocations as files named by IP address under /var/lib/cni/networks/NETNAME/. Each file contains the container ID that owns the allocation, making it straightforward to audit or debug stale allocations after a crash.

When to use it

• Debugging why a container cannot reach an external service when the host can, requiring inspection of iptables FORWARD and POSTROUTING rules and the docker0 bridge ARP table
• Designing multi-tenant isolation on a bare-metal host where different customer workloads must share a NIC but not see each other's traffic, using network namespaces, VLAN subinterfaces, and policy routing tables
• Tracing intermittent connection resets in a Kubernetes cluster that turn out to be conntrack table exhaustion causing SYN packets to be dropped by the stateful FORWARD chain
• Implementing a custom CNI plugin for a bare-metal environment where no existing plugin satisfies the required IP allocation or BGP peering model
• Tuning ECMP routing on a host with multiple uplinks to achieve per-flow load balancing without packet reordering, using ip route multipath and sysctl rp_filter settings
• Auditing iptables rules on a production node after a security incident to identify unauthorized DNAT or SNAT rules that redirect traffic to attacker-controlled endpoints
• Setting up a Linux router between two subnets using ip_forward, static routes, and NAT masquerade during a lab or interview live-coding exercise
• Diagnosing ARP thrashing on a host with bonded interfaces where gratuitous ARPs cause MAC table flapping on the upstream switch

Interview questions

• Trace a TCP SYN packet from a Docker container to the internet through every iptables chain it visits.
• Explain veth pair mechanics at the kernel level. What happens when you write a byte to one end?
• What is the difference between PREROUTING and POSTROUTING in iptables? When would you use each?
• How does ARP work for pod-to-pod traffic across different Kubernetes nodes? Walk through the full resolution path.
• Describe the CNI plugin contract precisely. What must a plugin do on ADD, and what guarantees must it make on DEL?

More Linux topics

• LDAP & sssd Integration
• Linux Audit System
• Linux Boot Process
• Linux Capabilities
• Linux Filesystem
• Linux Kernel & /proc
• Linux Networking Tools
• Linux Processes
• Linux Signals
• Linux Storage
• lsof: List Open Files
• LUKS & dm-crypt Encryption