strace & ltrace

Tracing system calls and library calls to debug mysterious application behavior.

Overview

Overview strace intercepts every interaction between a process and the Linux kernel. Since all I/O, file access, memory allocation (ultimately), and process management goes through system calls, strace reveals exactly what a program is doing at the lowest level.

How strace works: strace uses the ptrace() system call to attach to a process and intercept each syscall entry and exit. This introduces significant overhead (typically 10-100x slowdown) — use on production with caution, and use -c for summary first.

Basic usage: - strace command: traces from process start - strace -p PID: attach to a running process - strace -f: follow child processes created by fork() - strace -ff: follow forks and write each to a separate file (for threaded apps)

Output format: syscall_name(arguments) = return_value. On error: syscall_name(arguments) = -1 ERRNO (error description). Common errno: ENOENT (No such file or directory), EACCES (Permission denied), EAGAIN (Resource temporarily unavailable / try again), ECONNREFUSED (Connection refused), ETIMEDOUT (Connection timed out).

Filtering: - -e trace=file: only file-related syscalls - -e trace=network: only network syscalls (socket, connect, recv, send) - -e trace=process: process management (fork, execve, wait) - -e trace=memory: memory operations (mmap, brk, mprotect) - -e trace=open,read,write: specific syscalls

Performance analysis: - -c: count syscalls and show summary with time, calls, errors — great first step - -T: show time spent in each syscall - -tt: absolute timestamps with microseconds

ltrace: Similar to strace but for library function calls. ltrace command shows calls to shared library functions (malloc, free, fopen, strcmp, pthread_mutex_lock, etc.). Less overhead than strace for I/O-heavy workloads.

Key concepts

strace -p attach
strace -p PID attaches to a running process without restarting it. -f follows forks. Output shows syscall(args) = return or ERRNO. Ctrl+C detaches without killing the process.
-e trace categories
-e trace=file for filesystem ops, =network for sockets, =process for fork/exec, =memory for mmap. Combine: -e trace=file,network. Dramatically reduces output volume.
-c for summary stats
strace -c p PID sleeps N — shows count, time, and errors per syscall after detaching. Find where most time goes without reading thousands of lines of output.
ENOENT/EACCES/EAGAIN
ENOENT: file/dir not found at exact path shown. EACCES: permission denied (check ownership and mode). EAGAIN: non-blocking resource not ready (normal in event loops). ECONNREFUSED: port not listening.

When to use it

  • Debugging "file not found" when the file seems to exist
  • Understanding why an application is slow (where is it spending time?)
  • Finding which network connections an application is making
  • Diagnosing permission denied errors with complex file paths

Interview questions

  • An application is mysteriously slow. How would you use strace to find where it's spending time?
  • A program fails with "No such file or directory" but the file exists. How does strace help you debug this?

Sign up free to read the full topic

More Linux topics

Start free on Cariara · Coding problems · System design studio