Native Debugging — gdb, Core Dumps and Symbols

Getting an actionable backtrace out of a crash you did not witness — gdb on optimized builds, core dump capture in containers and CI, separate debug info, build-ids, and debuginfod.

Overview

From "it crashed" to "this line, this value"

This topic closes the gap between "the service crashed" and "here is the line that crashed and the value that caused it". Most teams lose days there, because the artifacts that close it must be arranged before the crash.

Three independent prerequisites

A core is written — RLIMIT_CORE, /proc/sys/kernel/core_pattern naming a writable destination, a non-setuid process; in containers, the host pattern reachable from the container, since core_pattern is not namespaced — Cores are disabled by default on most distributions
The exact binary is retrievable — A core records that a library was mapped at an address, not its contents — Container images are built and thrown away
Matching debug info is findable — Build-id — Binaries are stripped without archiving the .debug half

> [!WARNING] Skip these and a production crash produces a signal number and nothing else.

gdb on optimized builds

Breakpoints, watchpoints, bt, frame, info locals and thread apply all bt are familiar. A senior answer is comfortable at -O2: is not a debugger failure but DWARF honestly reporting the value does not exist there.

Edit-compile-debug cycle — -Og, the documented answer
Production -O2 binary you cannot rebuild — Read registers, know inlined frames have collapsed, distrust a backtrace that looks too clean

Symbol management is where the leverage is

Set DEBUGINFOD_URLS and set debuginfod enabled on and the filing problem becomes an HTTP GET.

> [!TIP] An internal debuginfod over your release artifacts is a small service that eliminates a whole category of incident-time archaeology.

CI is what distinguishes a candidate

A crashing test reports only an exit code — Cores enabled, gdb in batch mode on failure, and the core, binary, debug file and backtrace uploaded as artifacts

> [!IMPORTANT] That configuration converts every flaky crash into evidence. It is the difference between a team that fixes intermittent crashes and one that reruns the job.

Key concepts

core_pattern: /proc/sys/kernel/core_pattern controls where cores go. Supports %p and %P for PID in the crashing and initial namespaces, %e and %E for executable name and path, %t timestamp, %h hostname, %s signal, %u and %g for UID and GID. A leading pipe character sends the dump on stdin to an absolute-path program running as root in the initial namespace, which is how systemd-coredump works.
coredump_filter: /proc/PID/coredump_filter is a bitmask selecting which mappings enter the core: bits 0-3 for anonymous and file-backed, private and shared; bit 4 for ELF headers; bits 5-6 for private and shared huge pages; bits 7-8 for DAX. Default is bits 0, 1, 4, and 5. Widening it makes cores enormous; narrowing it can omit the mapping you needed.
build-id: A unique hash in the .note.gnu.build-id section, emitted with -Wl,--build-id, present identically in the executable and its separated debug file, and recorded in the core for every mapped object. It is what makes "no matching build-id" a loud error rather than a silently wrong backtrace, and it is the key debuginfod and /usr/lib/debug/.build-id lookups use.
Separate debug info: objcopy --only-keep-debug extracts DWARF into a .debug file, strip -g removes it from the shipped binary, and objcopy --add-gnu-debuglink= records the association with a CRC-32. gdb searches /usr/lib/debug/.build-id/xx/rest.debug first, then the debuglink name beside the binary, then a .debug subdirectory, then under /usr/lib/debug plus the binary path.
debuginfod: An HTTP server distributing ELF, DWARF, and source files by build-id, shipped with elfutils since 0.178. Configure clients with DEBUGINFOD_URLS and enable in gdb with set debuginfod enabled on. gdb then fetches missing debug info, executables, and source on demand — the practical replacement for manually locating symbols during an incident.
-Og: GCC optimization level documented as "the optimization level of choice for the standard edit-compile-debug cycle," offering a blend of optimization, fast compilation, and debugging experience. Enables variable tracking and disables passes that most damage debuggability. It is not a substitute for being able to reason about an -O2 binary you cannot rebuild.
thread apply all bt: The first command to run on any multithreaded core. Prints a backtrace for every thread, revealing deadlocks (two threads each blocked in a lock acquisition), the thread that actually took the signal, and threads stuck in syscalls. The full variant adds local variables per frame and is what you want in a CI artifact.
Minidump and PDB: The Windows equivalents. MiniDumpWriteDump writes a size-constrained subset of process state, from a normal minidump up to MiniDumpWithFullMemory. Reading one requires both the binaries and matching PDB files; symbols match by GUID and age rather than build-id, with a symbol server and _NT_SYMBOL_PATH playing debuginfod role.

When to use it

A service crashes in production or staging and you cannot reproduce it locally, so the core file is the only evidence that will ever exist
A test crashes intermittently in CI and currently produces only a signal number and an exit code
You ship stripped binaries and need a symbol strategy that keeps them debuggable without shipping DWARF to customers
A backtrace looks implausible — missing frames, wrong function names — and you need to determine whether the symbols actually match the binary
You are moving to containers and need to re-establish core capture, which silently stops working because core_pattern is a host setting

Interview questions

A production service segfaulted and you have a core file. Walk through getting from that to a root cause.
Explain separate debug info and build-ids. Why strip production binaries at all if you need symbols?
How do you capture core dumps from a containerized service, and what changes versus a bare-metal host?
You are debugging an -O2 build and half the locals show . How do you proceed, and what does -Og change?
How do you make a crashing test in CI produce an actionable backtrace instead of an exit code?