Sanitizers and Memory Debugging

AddressSanitizer, UBSan, ThreadSanitizer and MemorySanitizer — what each instruments, what it costs, and where each belongs in a CI pipeline. Includes Valgrind memcheck as the no-rebuild fallback and how to read a sanitizer report.

Overview

Delayed, misattributed failures

Memory bugs survive code review and unit tests, then crash in production with a stack nowhere near the fault. A heap overflow corrupts a neighbor and the failure surfaces later in unrelated code; a use-after-free reads recycled memory whose values look plausible until they do not. Sanitizers turn these into immediate, precise reports at the exact instruction that did the wrong thing.

Compiler instrumentation, not attachment

-fsanitize=address makes clang or gcc rewrite loads and stores to check a shadow table, and links a runtime that owns the allocator and reporting.

Coverage is per translation unit — A prebuilt vendor .a is uninstrumented: its bugs are invisible, and bugs surfacing through it may be reported at the wrong boundary
Changing sanitizers means a full rebuild in a separate build directory — Sanitizer CI is a matrix of build configurations, not a flag on an existing artifact

The four sanitizers, plus Valgrind

AddressSanitizer — Heap, stack and global overflow, use-after-free, use-after-return, use-after-scope, double and invalid free; bundles LeakSanitizer — About 2x time, 3x memory
UndefinedBehaviorSanitizer — Cheap independent checks for UB the optimizer may exploit — Small
ThreadSanitizer — Data races stress tests cannot reliably reproduce — 5x to 15x
MemorySanitizer — Reads of uninitialized memory — Whole program instrumented, standard library included
Valgrind memcheck — No rebuild: last resort for a binary you did not compile, and the only one usable on a production artifact — 10x to 30x; sees only the heap, so no stack or global overflows

Placement is the judgment being tested

Sanitizers only find bugs on paths your tests execute, so a sanitizer build with a weak suite finds nothing. ASan plus UBSan on every PR: tolerable cost, high yield. TSan nightly: expensive, and needs the whole dependency graph rebuilt. Fuzzing under ASan multiplies both: the fuzzer generates inputs, the sanitizer turns silent corruption into a recordable crash.

Have you actually read a report?

fd in the shadow dump — Freed heap region
Second stack — The free site
Third stack — The allocation site
Access 8 bytes inside a 20-byte region — Use-after-free, not overflow

> [!TIP] Anyone can enable a flag. Reading these details is what separates someone who ran a sanitizer once from someone who debugs with one.

Key concepts

Shadow memory: A parallel region where every aligned 8-byte granule of application memory maps to one byte at (addr >> 3) + offset. The byte records how many of the eight bytes are addressable, or a poison code naming why they are not. Every instrumented load and store consults it before executing.
Redzone: Poisoned padding that ASan inserts around heap allocations, between stack locals, and after globals. An off-by-one access lands in a redzone and is reported immediately, instead of silently corrupting the neighbouring object and crashing somewhere unrelated later.
Quarantine: Freed heap blocks are withheld from reuse and left poisoned so a subsequent access reports heap-use-after-free rather than succeeding on recycled memory. Bounded by ASAN_OPTIONS=quarantine_size_mb; a smaller quarantine saves RSS in long soak tests but lets old use-after-free bugs slip through.
Recoverable vs non-recoverable check: UBSan checks are recoverable by default: they print and continue, and the process exits zero. -fno-sanitize-recover=all makes them abort, and -fsanitize-trap= replaces the runtime call with a trap instruction entirely. Without one of these, UBSan findings never fail CI.
Instrumentation completeness: ASan tolerates uninstrumented code with reduced coverage. TSan does not — it requires all code compiled with -fsanitize=thread, because a lock taken in an uninstrumented library is invisible and produces false race reports. MSan is stricter still and needs an instrumented libc++.
Suppressions vs ignorelist: Two different mechanisms. A suppressions file is runtime (ASAN_OPTIONS=suppressions=, LSAN_OPTIONS=suppressions= with leak: entries, TSAN_OPTIONS with race:/thread:/called_from_lib: entries) and silences reports. A sanitizer special case list is compile time and skips instrumenting the named src, fun, global or type in the first place.
LeakSanitizer: A leak detector that runs at process exit under ASan on Linux, and also exists standalone as -fsanitize=leak without the 2x ASan overhead. Controlled by ASAN_OPTIONS=detect_leaks; default behaviour varies by platform, so verify rather than assume it is on.
Origin tracking: MSan option -fsanitize-memory-track-origins=2 that records where an uninitialized value was created, not just where it was consumed. Adds 1.5x-2x on top of MSan baseline and is the difference between a report you can act on and one you cannot.

When to use it

Every pull request: a dedicated -fsanitize=address,undefined build running the unit test suite, with -fno-sanitize-recover=all so UBSan findings actually fail the job
Nightly or pre-release: a separate -fsanitize=thread build for data races, since TSan cannot share a binary with ASan and costs 5x-15x
Fuzzing: libFuzzer or AFL++ targets always built with ASan and UBSan, because a fuzzer without a sanitizer only finds inputs that already crash
Debugging a specific crash: MSan with -fsanitize-memory-track-origins=2 when the symptom is nondeterministic behavior that smells like an uninitialized read
Third-party or release binaries you cannot rebuild: Valgrind memcheck with --leak-check=full --track-origins=yes

Interview questions

Walk me through this AddressSanitizer report line by line. What is the bug, and what do the shadow bytes tell you?
Why can ASan and TSan not be combined, and how does that shape your CI matrix?
Your ASan and UBSan job has been green for six months, but a customer just hit a signed overflow bug in code that job exercises. What went wrong?
When would you use Valgrind memcheck instead of a sanitizer, and what does it cost you?
How does ASan actually detect a heap buffer overflow, and what classes of memory bug does it structurally miss?