journalctl & Log Management

journalctl filtering, persistent logs, vacuum, priority levels, and journald.conf tuning.

Overview

Overview journald is systemd's logging daemon. It collects log entries from the kernel, systemd units, and any process that writes to stdout/stderr or calls syslog(). Unlike traditional text-based syslog, journald stores logs in a structured binary format that supports efficient filtering by time, unit, priority, PID, UID, and arbitrary fields.

Storage Locations

Volatile (default on many distros): /run/log/journal/ — lost on reboot. Kernel events still survive if the machine reboots cleanly because the EFI pstore or pmsg captures crash logs.

Persistent: /var/log/journal/ — survives reboots. Enabled by setting Storage=persistent in /etc/systemd/journald.conf or by creating the directory: mkdir -p /var/log/journal && systemd-tmpfiles --create --prefix /var/log/journal.

Priority Levels

Journald uses syslog priority numbers:

emerg — 0 — System is unusable
alert — 1 — Action must be taken immediately
crit — 2 — Critical conditions
err — 3 — Error conditions
warning — 4 — Warning conditions
notice — 5 — Normal but significant
info — 6 — Informational
debug — 7 — Debug-level messages

journalctl -p err shows messages at level 3 (err) and above — everything more severe.

Output Formats

journalctl -o FORMAT supports: short (default), verbose (all fields), json (one JSON object per line), json-pretty, cat (message only), export (for piping to another journal).

Key concepts

journald: systemd logging daemon that collects from all units. Binary format at /run/log/journal/ (volatile) or /var/log/journal/ (persistent). Supports structured fields for rich filtering.
journalctl -u: Filter to a specific systemd unit. journalctl -u nginx.service -f follows nginx logs in real time. Combine with --since for time-bounded queries.
-p priority: Filter by syslog priority. -p err shows err(3) and above (err, crit, alert, emerg). -p warning..err shows a range. Lower number = higher severity.
--vacuum-size / --vacuum-time: Prune old journal data. journalctl --vacuum-size=1G keeps only 1 GiB of logs. journalctl --vacuum-time=30d deletes logs older than 30 days.
Storage=persistent: Set in /etc/systemd/journald.conf. Writes journal to /var/log/journal/ instead of /run/log/journal/. Required for logs to survive reboots.
-o json: Output each log entry as a JSON object with all structured fields. Useful for piping to jq or ingesting into a log aggregator. -o json-pretty for human-readable.

When to use it

Tailing a specific service log — journalctl -u nginx.service -f
Investigating an incident at a specific time — journalctl --since "2024-01-15 14:00" --until "2024-01-15 16:00"
Finding all errors across all services — journalctl -p err --since today
Checking disk usage and rotating old logs — journalctl --disk-usage then --vacuum-size or --vacuum-time
Making logs persistent across reboots — Storage=persistent in journald.conf

Interview questions

How do you make systemd journal logs persist across reboots and manage their disk usage?
Show common journalctl query patterns for production incident investigation.