Logging Framework

Customizable Logger

Design the logging library an application calls: named loggers with levels, handlers that write to different places, pluggable formats, and a queue so slow destinations never stall the caller.

Overview

An application calls log.info("order placed", order_id=42) and moves on. Behind that one line a library decides whether the message matters, turns it into text or JSON, and writes it to a console, a file, or a remote service, from many threads at once, without slowing the application down. You are designing that library.

The problem is popular because it is small enough to code in an hour yet touches most of the design vocabulary interviewers want to hear: a pluggable format, a pluggable destination, a tree of named loggers with inherited levels, a lock around I/O, and a queue with a worker thread. It also has real reference designs: Python's logging module and Java's Log4j 2 share the same broad shape (loggers, levels, handlers or appenders, formatters or layouts), and the code below follows it so you can point at it.

Scoping the problem

Which levels, and can a handler have a level of its own? — Filtering happens twice: at the logger and at each handler — DEBUG, INFO, WARNING, ERROR, CRITICAL; yes
One global logger or named loggers? — Named loggers let you turn up one module without flooding the rest — Named, in a dotted tree that inherits levels
Which destinations? — Each is a class; the framework must not care which — Console and file now; a network sink later
Text or structured output? — A formatter is a strategy; both must be possible — Both, chosen per handler
Must logging never block the caller? — Adds a queue and a worker thread, and a policy when the queue is full — Yes for slow sinks
Many threads? — A lock per handler around the write — Yes
What if a destination fails? — A broken disk must not crash the app — Count the error and keep going

The pipeline

A record passes through two filters and one transformation before it is written. The logger's effective level drops most records early and cheaply. Each handler then applies its own level, formats the record with its own formatter, and writes it.

Levels

Levels are integers so that filtering is one comparison. The values below are the ones Python's logging module uses, spaced by ten so custom levels fit between them.

DEBUG — 10 — Detail useful only while diagnosing a problem
INFO — 20 — Normal events worth a line: a request served, a job finished
WARNING — 30 — Something unexpected that the program handled
ERROR — 40 — An operation failed
CRITICAL — 50 — The program cannot continue

A record is written when its level is at or above the threshold, so a handler set to ERROR writes ERROR and CRITICAL and nothing else.

A tree of named loggers

get_logger("app.db") returns a logger whose parent is app, whose parent is the root. The same name always returns the same object, so modules never pass loggers around. Two things flow through the tree.

Level — Down, from ancestors — A logger with no level of its own uses the nearest ancestor that has one; the root always has one (WARNING by default)
Records — Up, to ancestors' handlers — After its own handlers, a logger hands the record to its parent's handlers, and so on to the root, unless propagate is off

This is how you turn on DEBUG for one noisy module in production: set the level on app.db and leave the rest alone. The propagation rule is also why an application configures handlers on the root once and every module's logger reaches them.

Thread safety and a slow destination

Two threads writing to one file at the same time interleave their bytes. Each handler therefore owns a lock and takes it around emit. The level check and the formatting run outside the lock, because they touch no shared state. Python's logging docs describe the same split: one lock for the module's shared data and one lock per handler for its I/O.

A lock makes writes correct but still makes the caller wait for the disk or the network. AsyncHandler fixes that: callers put the record on a bounded queue and return; one worker thread takes records off the queue and hands them to the wrapped handler. Python ships this as QueueHandler plus QueueListener; Log4j 2 calls it AsyncAppender, with a queue of 1,024 events and blocking behavior by default.

When the queue is full, something has to give. Say the options out loud.

Block the caller (default here and in Log4j 2) — Slows down until the worker catches up — Nothing lost
Drop the record and count it — Never slows down — Gaps; the counter tells you how many
Write to a fallback handler — Never slows down — Complete, but split across two places

What you will be able to do

Draw the pipeline: logger level, record, per-handler level, formatter, destination.
Explain the logger tree, inherited levels and propagation, and use them to turn up one module.
Put the lock in the right place: around the write, not around the level check.
Add a queue and a worker thread, and name the three policies for a full queue.
Keep a broken destination from crashing the application.

Functional requirements

Five levels, DEBUG to CRITICAL, each an integer; a record is written when its level is at or above the threshold
Named loggers in a dotted tree; a logger without its own level inherits from the nearest ancestor that has one
Any number of handlers per logger; a record reaches the handlers of the logger and of its ancestors unless propagation is turned off
Each handler has its own level and its own formatter
Two formats: a text line and a JSON document; both carry structured fields passed as keyword arguments
Destinations: a stream (console), a file, memory (for tests), and an asynchronous wrapper around any of them
A simple call site: log.info(message, key=value)

Non-functional requirements

Safe for many threads: a lock per handler around the write, none around the level check
A record below the effective level costs a level lookup and one comparison; no record is built and no handler runs
A slow destination behind AsyncHandler never blocks the caller unless the queue is full and blocking is chosen
A handler that raises counts the error and does not crash the application or stop the other handlers
Adding a destination or a format is one new subclass; nothing existing changes

Interview questions

Why does a handler have its own level when the logger already has one?
Walk me through the asynchronous handler. What can go wrong?
Why not a single global logger, a Singleton?
How would you support structured logging?
Where exactly do you need locks, and why not one global lock?
How do you test a logging library without reading files or the console?