ftrace & bpftrace Tracing

Kernel function tracing with ftrace, event tracing, bpftrace one-liners for production observability without kernel modules.

Overview

ftrace

ftrace is the Linux kernel's built-in tracing framework, accessible through the tracefs virtual filesystem at /sys/kernel/debug/tracing/ (or /sys/kernel/tracing/ on modern kernels).

Key ftrace Tracers

function — Traces every kernel function call — very high overhead
function_graph — Traces entry and exit with call graph indentation
nop — Disables tracing (default)
blk — Block I/O events

Basic ftrace Usage

trace-cmd (ftrace frontend)

bpftrace

bpftrace is a high-level scripting language for eBPF tracing. Programs are compiled to BPF bytecode at runtime.

Probe Types

kprobe — kprobe:tcp_connect — Kernel function entry
kretprobe — kretprobe:tcp_connect — Kernel function return
tracepoint — tracepoint:syscalls:sys_enter_read — Stable kernel events
uprobe — uprobe:/bin/bash:readline — User-space function
profile — profile:hz:99 — CPU sampling

Key concepts

tracefs: Virtual filesystem mounted at /sys/kernel/debug/tracing/. ftrace is controlled by reading/writing files: current_tracer, set_ftrace_filter, tracing_on, trace_pipe.
function_graph tracer: Traces kernel function entry and exit with indentation showing the call graph. Shows duration of each call. Use with set_graph_function to filter.
kprobe vs tracepoint: kprobes attach to arbitrary kernel functions (fragile — break across versions). Tracepoints are stable kernel interfaces with stable argument names — prefer tracepoints for long-lived scripts.
bpftrace @maps: BPF maps used to aggregate data. @[key] = count() builds a frequency map. @var = hist(value) builds a power-of-2 histogram. Printed on Ctrl+C.

When to use it

Tracing kernel function calls to debug an obscure I/O or networking issue without rebooting
Writing custom observability that captures exactly the event and data needed
Measuring syscall latency for a specific process in production without code changes

Interview questions

How would you trace what system calls a specific process is making in production without using strace?