Thread Pools

Why pools exist, how Java's ThreadPoolExecutor decides between a new thread, the queue and rejection, how to size a pool from the wait-to-compute ratio, and a small pool built and tested in Python.

Overview

A thread pool keeps a fixed set of worker threads and a queue of tasks. Callers submit tasks; idle workers take them from the queue and run them; results come back through a future. The threads live for the life of the pool instead of one per task.

A pool buys three things. Creating and destroying a thread for every task costs time and memory, and the pool pays it once. The number of workers caps how much runs at once, so a burst of requests cannot create thousands of threads. And the queue gives overload a place to show up, as a growing queue, where you can measure it and push back.

The shape of a pool

In Python the standard pool is concurrent.futures.ThreadPoolExecutor. Its submit returns a Future; result(timeout) waits for the value and re-raises the task's exception if it failed; using the executor in a with block shuts it down and waits for pending work. Since Python 3.8 the default max_workers is min(32, os.cpu_count() + 4) (3.13 uses os.process_cpu_count()).

How Java's ThreadPoolExecutor decides

ThreadPoolExecutor has a core size, a maximum size and a queue. Its docs give the rule for each new task, and it surprises people: extra threads beyond the core are created only when the queue is full.

The queue you choose changes the behavior completely.

SynchronousQueue (direct handoff) — Holds nothing; every task needs a free thread or a new one — Threads grow without bound if the maximum is unbounded
Unbounded LinkedBlockingQueue — Never more than core threads; the maximum has no effect — The queue grows without bound under overload
Bounded ArrayBlockingQueue — Queue, then grow to the maximum, then reject — Needs tuning of queue size against pool size

Executors.newFixedThreadPool(n) uses a shared unbounded queue, and newCachedThreadPool creates threads as needed and drops ones idle for sixty seconds. Both are fine for tests and dangerous on a request path, for the reasons in the Risk column. For production, build the ThreadPoolExecutor yourself with a bounded queue and a chosen rejection policy.

Sizing a pool

For work that only computes, more threads than cores just adds switching; about one per core is the ceiling. For work that waits, a core is idle while a thread waits, so more threads keep it busy. If each task computes for C and waits for W, one core stays busy with about (W + C) / C threads, so a starting size is cores × (1 + W / C).

Pure CPU (parsing, hashing) — 8 — 0 — 10 ms — 8 × (1 + 0) = 8
Mostly waiting (HTTP calls) — 8 — 90 ms — 10 ms — 8 × (1 + 9) = 80

A second check comes from the arrival rate, by Little's law: tasks in flight equal arrival rate times time per task. At 400 requests per second and 100 ms each, about 400 × 0.1 = 40 tasks are in progress at once on average; a pool smaller than that queues, and one much larger sits idle. Both numbers are starting points; measure queue depth, worker utilization and latency, then adjust.

Threads are not free: every Java platform thread has its own stack, 1 MB by default on Linux x64 (-Xss), so 200 threads set aside about 200 MB of stack address space. A downstream limit also caps the useful size: 80 workers sharing a database pool of 20 connections means 60 of them wait on the pool.

Pool deadlock: a task waiting on its own pool

If a task submits another task to the same pool and waits for it, and every worker is doing the same, nobody is left to run the inner tasks. The Python docs warn about exactly this. The program shows it with one worker and a timeout.

What you will be able to do

Explain why pools exist: reuse, a cap on concurrency, and a visible queue.
Trace what ThreadPoolExecutor does with a new task, and pick the queue type.
Size a pool from cores and the wait-to-compute ratio, and check it with Little's law.
Choose a rejection policy and shut a pool down cleanly.
Spot pool deadlock and fix it with separate pools.

Key concepts

Worker reuse
Bounded concurrency
Task queue
Futures
Core and maximum size
Rejection policy
Pool sizing
Graceful shutdown

Interview questions

How do you size a thread pool?
What happens when a thread pool is full? Compare the rejection policies.
How do you shut down a thread pool cleanly?
Implement a simple thread pool.