NAT Gateway Port Exhaustion Under Bursty Lambda Traffic

A burst of Lambda invocations exhausted NAT Gateway SNAT ports, causing connection failures to external APIs with confusing "connection refused" errors.

Overview

A tripled nightly batch pushed 800 Lambdas through one NAT Gateway to one external IP, ran it out of source ports, and cost 90 minutes of debugging the wrong system.

What happened

The pipeline ran Lambda functions in a VPC that called third-party APIs in nightly batches. It had been stable for months until a customer acquisition campaign changed the load overnight.

Why it broke

NAT Gateway supports 55,000 simultaneous connections, but that was not the limit that mattered. Every connection shared one NAT IP and one destination IP, so they all drew from the same pool of source ports.

Impact

Concurrent Lambda executions — 800
Persistent HTTPS connections per Lambda — 3
NAT Gateway simultaneous connection limit — 55,000
Source ports from one NAT IP to one destination IP — 65,535
Time spent debugging the external API — 90 minutes

The fix

NAT Gateways — One, shared by all 800 Lambdas — Several, with Lambda subnets spread across them
Connections per Lambda — 3 persistent each — Capped per Lambda instance by pooling limits
First place to look — The external API — NAT Gateway error metrics

> [!WARNING] Port exhaustion looks exactly like the remote service being down. Check the NAT Gateway metrics before blaming the other side.

Key concepts

SNAT (Source NAT): Network Address Translation on the source port — NAT Gateway translates Lambda's private IP and port to a public IP and ephemeral port for outbound connections.
Port exhaustion: Running out of available source port numbers (max 65,535 per destination IP/port pair), preventing new TCP connections from being established.
TIME_WAIT state: A TCP connection state after close where the port is held for 2x maximum segment lifetime (typically 60-120 seconds), effectively consuming a port slot.
VPC endpoint: A private connection to AWS services that bypasses the NAT Gateway entirely, reducing both cost and SNAT port consumption for AWS API calls.

When to use it

Designing Lambda networking for high-concurrency workloads in VPCs
Explaining NAT Gateway limits and port exhaustion symptoms
Justifying VPC endpoint usage for AWS service calls from Lambda

Interview questions

A Lambda function in a VPC is failing with connection errors to an external API at scale. How do you diagnose NAT Gateway port exhaustion?