S3 High Latency Spikes
Diagnose S3 request latency spikes and design prefix strategies to avoid throughput limits.
Overview
Overview
S3 latency spikes have several distinct root causes, each with a different fix. Before optimizing, separate the two latency components that S3 CloudWatch reports: FirstByteLatency and TotalRequestLatency.
Diagnosis flow
- FirstByteLatency — S3-side issues
- TotalRequestLatency with normal FirstByteLatency — Network or object size issues
Per-prefix request limits
- PUT/COPY/POST/DELETE — 3,500 requests per second
- GET/HEAD — 5,500 requests per second
A prefix is the key name up to the first delimiter, so all objects under logs/2024/ share one prefix budget. Past the limit, S3 returns 503 SlowDown.
LIST on hot paths
ListObjectsV2 is O(N): it scans every object under a prefix in lexicographic order. Listing a prefix with 10 million objects to find the latest file can take seconds.
- Pointer object — Store e.g. latest/manifest.json and update it on upload
- DynamoDB index — Index S3 object metadata in DynamoDB
> [!WARNING] Never use LIST on a hot path.
Large uploads and distant clients
- Multipart upload — Uploads parallel chunks and retries only the failed ones; without it, one network interruption fails the whole upload. The SDK does it for you with TransferManager in Java or the high-level upload APIs elsewhere — Recommended above 100MB, required above 5GB (the single PUT limit)
- S3 Transfer Acceleration — Routes uploads through the nearest CloudFront edge, then the AWS private backbone to the bucket — Uploads from distant geographic regions
- S3 Express One Zone — Consistent single-digit millisecond latency for high-throughput workloads, with single-AZ durability — Caches or scratch storage, not primary data
> [!TIP] Always capture the x-amz-request-id header from S3 responses: AWS Support uses it to trace a specific request if you open a case about unexplained latency spikes.
Key concepts
- FirstByteLatency
- CloudWatch S3 metric measuring time from when S3 receives the request to when it sends the first byte of the response. Reflects server-side processing time, excluding network transfer.
- Prefix Throughput Limit
- S3 limits request rates per prefix: 3,500 write and 5,500 read requests per second. Exceeding these returns 503 SlowDown errors and requires distributing objects across multiple prefixes.
- Multipart Upload
- S3 mechanism to upload large objects as parallel chunks. Enables resumable uploads, parallel throughput, and is required for objects over 5GB; strongly recommended above 25MB.
- S3 Transfer Acceleration
- Routes S3 uploads and downloads through CloudFront edge locations, then over AWS private backbone, reducing latency for geographically distant clients.
- S3 Express One Zone
- A high-performance S3 storage class offering sub-millisecond consistent latency and higher throughput at the cost of single-AZ durability. Suited for caches, ML training data, and scratch storage.
When to use it
- Diagnosing 503 SlowDown errors or latency spikes in S3-heavy applications
- Explaining S3 throughput limits and prefix sharding in system design interviews
- Designing high-throughput data pipelines that use S3 as a storage layer
Interview questions
- Your S3-based log storage pipeline starts returning 503 SlowDown errors when log volume triples. The key structure is logs/YYYY-MM-DD/server-id/filename. How do you fix this without changing the data model significantly?
Sign up free to read the full topic
More Troubleshooting topics
Start free on Cariara · Coding problems · System design studio