Connection Pool Exhaustion That Brought Down 12 Microservices

A single slow migration caused connection pool exhaustion on the RDS primary, cascading to 12 services that shared the same database.

Overview

One schema migration locked the orders table on a shared PostgreSQL instance, and within 4 minutes all 12 microservices had run out of database connections.

What happened

A database engineer ran ALTER TABLE orders ADD COLUMN delivery_estimate_v2 jsonb NOT NULL DEFAULT '{}' during a low-traffic window. Adding a column with a default rewrote all 300 million rows (PostgreSQL before 11 rewrites the table for this), holding a table-level lock for 18 minutes.

How it spread

Every query that touched orders queued behind the lock, and each waiting query held a pool slot. The services still looked healthy because their health checks used a bypass route.

RDS max_connections — 500
Total pool max across 12 services — 420 (84% of capacity under normal load)
Table size — 300 million rows
Lock duration — 18 minutes
Time to exhaust all pools — Within 4 minutes
Migration killed at — 22 minutes
Pool drain after the kill — 8 minutes
Total user impact — 27 minutes

The fix

Killing the migration released the lock, and the pools recovered on their own as the queue cleared.

orders table — Locked — Readable and writable
Service pools — Full of waiting connections — Drained over 8 minutes
Order operations for users — Failing — Recovered

> [!WARNING] A low-traffic window does not make a table lock safe. On a shared database, one lock reaches every service that touches the table.

Key concepts

Table-level lock: A lock that prevents all reads and writes to a table during a schema change (like adding a NOT NULL column without a default, or adding an index without CONCURRENTLY).
Online schema change: Techniques (pt-online-schema-change, pg_repack, GitHub gh-ost, or PostgreSQL CONCURRENTLY options) that perform DDL without holding table locks for the full migration duration.
Connection pool sizing: Choosing pool sizes that sum to less than the database max_connections limit, with headroom for admin connections and spikes.
PgBouncer: A connection pooler for PostgreSQL that multiplexes application connections onto a smaller set of actual database connections, raising the effective connection capacity.

When to use it

Explaining why ALTER TABLE on large tables requires lock-free migration strategies
Discussing connection pooling architecture for microservices sharing a database
Justifying PgBouncer or RDS Proxy for connection management

Interview questions

How do you safely add a column to a 300-million-row PostgreSQL table without downtime?