eBPF Programming

Overview

Overview

eBPF, originally Extended Berkeley Packet Filter, has grown from packet filtering into a general-purpose kernel execution environment. It lets you instrument and enforce policy inside the kernel without the risk or maintenance burden of a kernel module.

The 1990s Berkeley Packet Filter gave tcpdump a safe way to pick which packets to capture. The extended version added a 64-bit register file, arbitrary helper calls, persistent maps, and dozens of hook points across networking, tracing and security.

Safety: the verifier

Every program submitted through the bpf() syscall is validated before it may run. The verifier explores every path in the control-flow graph and tracks the type and value range of every register and stack slot.

• Reads out-of-bounds memory
• Calls a non-approved helper
• Loops without a provable bound

> [!IMPORTANT] A rejection comes with a human-readable log naming the failing instruction. eBPF is safer than a loadable module yet faster than user space, because there is no syscall boundary on the hot path.

Portability: CO-RE

Compile Once Run Everywhere ends the BCC-era need to ship Clang and kernel headers to every host for load-time recompiles.

The same binary runs on a 5.10 LTS kernel with one struct layout and a 6.6 kernel with another, with no compiler or headers on the target.

Layers of abstraction

• Raw bpf() syscall — Hand-assemble or compile instructions, manage every map fd — Maximum control
• libbpf — Write C, compile with Clang for BPF, use the typed skeleton API — The canonical middle layer, production agents
• BCC — Python or Lua on top, compiling BPF C at runtime — Interactive development; impractical on hosts without kernel headers
• bpftrace — A DSL that compiles to BPF and prints aggregates — One-liners in live debugging

Lifecycle: pinning and links

• Pinning to /sys/fs/bpf/ — Keeps a program or map alive after the loader exits, essential for in-place daemon upgrades
• Link objects, kernel 5.7 — Reference-counted attach handles that clean up when the fd closes, so no manual detach in signal handlers

> [!TIP] These lifecycle details are what make a production agent reliable.

Key concepts

eBPF Verifier and Safety Model

The verifier is a static analysis pass that runs inside the kernel before any BPF instruction executes. It performs a depth-first traversal of the program's control-flow graph, tracking the abstract state of every register and every byte of the stack frame at each instruction. Register state includes the type, which may be scalar, pointer to a map value, pointer to the packet data, or pointer to the context struct, and for scalars the minimum and maximum possible value. The verifier enforces several classes of rules. Pointer arithmetic is only allowed when the resulting pointer can be proven to stay within a known-safe memory region. A packet pointer offset by a runtime value is only safe if the program has already checked that the offset plus the access size is less than the packet length. Map value pointers require a null check after lookup before they can be dereferenced. Helper calls are only valid for the specific program type, for example XDP programs cannot call helpers that require an sk_buff. Loop support was historically forbidden entirely because the verifier could not prove termination for back-edges. Kernel 5.3 introduced bounded loop support: loops are allowed when the verifier can determine that the loop variable is a scalar with a finite range, meaning every back-edge has a decreasing bound. Kernel 5.17 added open-coded iterators for BPF_FOR_EACH_MAP_ELEM and similar patterns. When the verifier rejects a program, loading it via libbpf prints the verifier log to stderr. Reading this log is the primary debugging technique. A typical rejection message looks like the following. This tells you register r6 holds the context pointer but you attempted to read it as a frame pointer, meaning you accidentally used the wrong register. The log prints the source line that maps to the failing instruction, the BPF assembly, and the expected versus actual type.

CO-RE and BTF Portability

CO-RE, Compile Once Run Everywhere, solves the kernel version portability problem without requiring kernel headers or Clang on the target machine. The mechanism depends on BTF, BPF Type Format, a compact debug-info format embedded in the kernel itself and in eBPF object files. When you compile a CO-RE program with Clang, you use vmlinux.h generated from the running kernel's BTF by bpftool. Instead of including individual kernel headers that vary across versions you include this single generated header. Clang emits CO-RE relocations, special annotations in the BPF ELF section, wherever your program accesses a kernel struct field. Each relocation records the field name, the BTF type it belongs to, and the byte offset at compile time. When libbpf loads the object on the target kernel, it reads the kernel's BTF from /sys/kernel/btf/vmlinux, resolves the actual byte offset of the field in that kernel's struct layout, and patches the BPF instructions with the correct offset before calling the verifier. If the field does not exist in the target kernel, libbpf can optionally skip the program or substitute a default value using the BPF_CORE_READ_BITFIELD_PROBED family of helpers. A CO-RE field read looks like this. BPF_CORE_READ generates the CO-RE relocation. On a kernel where skc_dport is at offset 68 it patches offset 68; on a kernel where it moved to offset 72 it patches 72. The binary is identical in both cases.

Program Types and Hook Points

Each eBPF program type defines the context struct the program receives, the set of helpers it may call, and the return codes it may produce. Mixing types or calling wrong helpers causes verifier rejection. A kprobe program fires on entry to any kernel function. The context is struct pt_regs, and you use the PT_REGS_PARM1 through PT_REGS_PARM5 macros to read function arguments. A kretprobe fires on return and gives you PT_REGS_RC for the return value. A uprobe fires on a user-space function entry. You attach it to a binary path and a symbol offset. This does not require recompiling or instrumenting the target process. A tracepoint program fires on a stable kernel trace event defined with TRACE_EVENT. The context struct is generated from the event format file in /sys/kernel/debug/tracing/events/. Tracepoints are preferred over kprobes for production because their ABI is stable across kernel versions. An XDP program runs inside the NIC driver receive function. The context is struct xdp_md with data, data_end, and data_meta pointers. The return value must be XDP_PASS, XDP_DROP, XDP_TX, or XDP_REDIRECT. A TC program runs at the traffic control layer with a full struct __sk_buff context. It has access to more metadata than XDP but runs after sk_buff allocation. A cgroup/skb program fires on every packet sent or received by any socket in a cgroup, enabling per-container network policy. Sock_ops programs fire on individual TCP connection events such as connection establishment, setting socket options, or timeout.

Map Types

eBPF maps are the primary data structures for storing state between program invocations and for communicating with user space. Each map type is optimized for different access patterns. BPF_MAP_TYPE_HASH is a hash table keyed by arbitrary bytes. Lookup, update, and delete are O(1) average. It is the right choice for per-PID or per-IP counters. BPF_MAP_TYPE_ARRAY is a fixed-size array keyed by u32 index. It cannot be deleted from; update overwrites the existing slot. It is cache-friendly for small maps accessed by index. BPF_MAP_TYPE_LRU_HASH automatically evicts the least recently used entry when the map is full. Essential for connection tracking tables where the number of distinct keys is unbounded. BPF_MAP_TYPE_RINGBUF is the modern way to stream variable-length records to user space. The producer reserves a record, writes into it, and submits it atomically. The consumer polls with a file descriptor. It outperforms perf_event_array because it avoids per-CPU fragmentation and supports large records. BPF_MAP_TYPE_PERF_EVENT_ARRAY is the older per-CPU streaming mechanism. Each CPU has its own perf ring buffer. Preferred when targeting kernels older than 5.8 that lack ringbuf support. BPF_MAP_TYPE_PROG_ARRAY stores references to other BPF programs, enabling tail calls that replace the current stack frame, allowing programs that would exceed the 1 million instruction limit to be split across multiple programs.

libbpf Skeleton API

The libbpf skeleton is a code-generation workflow that removes all manual file descriptor management from eBPF user-space loaders. After compiling your BPF C file to an object, bpftool gen skeleton produces a header with typed structs for your maps and programs. A minimal counter daemon using the skeleton looks like the following. The open step memory-maps the BPF ELF and applies CO-RE relocations. The load step calls the verifier for each program section. The attach step creates the kprobe/tracepoint/XDP links. The destroy step closes all file descriptors and removes all links. The skeleton accessor skel->maps.syscall_count gives a typed bpf_map pointer without any string lookups. skel->progs.count_syscall gives the bpf_program pointer if you need to set program-level options like log verbosity before loading.

bpftrace One-Liners and Scripts

bpftrace is an awk-like front end that compiles to BPF internally. It is ideal for interactive debugging sessions where you want results in seconds without writing a C loader. Count system calls by process name. Measure block I/O latency as a power-of-two histogram. Trace every execve with arguments. Sample CPU stacks at 99 Hz across all processes for a flame graph. bpftrace variables starting with @ are BPF maps. The built-in aggregation functions count(), sum(), avg(), min(), max(), hist(), and lhist() map to BPF map operations. The BEGIN and END probes run setup and teardown logic. The limitation is that bpftrace scripts are not portable binaries; they recompile on each target and require bpftrace to be installed. For production agents use libbpf with CO-RE instead.

XDP Return Codes and Packet Processing

XDP programs return a verdict that tells the driver what to do with the frame. The four codes are XDP_PASS, XDP_DROP, XDP_TX, and XDP_REDIRECT. XDP_PASS hands the packet to the normal Linux network stack. Use this as the default action at the end of a firewall program after the deny rules have not matched. XDP_DROP discards the packet in the driver before sk_buff allocation. This is the fastest possible discard, measurable in nanoseconds per packet on modern NICs, and is used for DDoS mitigation and network policy enforcement. XDP_TX hairpins the packet back out the same interface. Useful for a reflective load balancer or for NAT on a single-leg network. XDP_REDIRECT sends the packet to a different destination. The destination can be another interface via bpf_redirect(), another CPU's XDP queue via bpf_redirect_map() with a CPUMAP, or a user-space AF_XDP socket via an XSKMAP for kernel-bypass receive. A minimal IP firewall using XDP looks like the following. Every pointer check is mandatory. The verifier rejects any access beyond data_end without a preceding bounds check on that specific pointer.

When to use it

• You need sub-microsecond packet filtering or load balancing without leaving the kernel, replacing iptables rules that become slow at scale with XDP programs that run before sk_buff allocation.
• You want zero-instrumentation observability: measuring syscall latency, tracking TCP retransmits, profiling CPU flame graphs, or auditing file opens by container, all without modifying application code or adding sidecars.
• You are building a Kubernetes CNI plugin or network policy engine and need per-packet enforcement at line rate across thousands of pods, as Cilium, Calico eBPF mode, and Antrea do.
• You need kernel-level security enforcement such as blocking specific syscall argument patterns, auditing privilege escalation, or sandboxing untrusted workloads, as Falco and Tetragon do.
• You want to profile production systems without restarting processes, using uprobes to instrument Go or Rust binaries at specific function entry and return points with nanosecond timestamps.
• You are replacing a BCC-based agent with a CO-RE libbpf binary so you can ship a single statically linked executable to heterogeneous kernel versions without shipping Clang or kernel headers.
• You want to implement socket-level load balancing or connection-level TCP metrics collection using sock_ops and sk_msg hooks that fire on individual connection events.

Interview questions

• What are the main constraints the eBPF verifier enforces, and how do you debug a verifier rejection?
• How does CO-RE achieve kernel portability without recompiling on the target, and what are its limitations?
• Write a libbpf kprobe program that counts how many times each process calls write(), with a user-space loader that prints the top counts every second.
• What is the difference between XDP and TC hook points, and when would you choose one over the other?
• Compare bpftrace with raw eBPF using libbpf. When is each the right tool, and what are the trade-offs?

More DevOps topics

• DORA 2024 — 39 Capabilities & AI Impact
• DORA Four Keys — Deep Dive
• DORA Four Keys — Implementation
• DORA Metrics
• DORA Metrics — Definitions, the Math, and How to Measure Them
• eBPF Observability
• Ephemeral and Preview Environments
• etcd and the API Server — The Write Path, Raft, and Why 3 vs 5 Members Matters
• Event-Driven Architecture
• Event-Driven Automation and StackStorm
• External Secrets Operator
• Feature Flags