Handling Large Blobs
How to upload, store, process and serve files from megabytes to terabytes: bytes in object storage and metadata in a database, direct uploads with presigned URLs, multipart and resumable uploads, and downloads through a CDN.
Overview
A blob (binary large object) is a file your system stores but does not look inside: a photo, a video, a PDF, a backup, a model checkpoint. Blobs are large compared with the rows of a database, often megabytes to gigabytes, and they are written once and read many times.
Blobs break the habits that work for small data. A 2 GB video should not travel through an API server, which would spend minutes and memory relaying bytes it does not need to read. It should not sit in a database row, where it bloats backups and replication. And its upload will be interrupted for some users, so starting over from zero is not acceptable.
The standard design separates bytes from metadata. The bytes go into object storage, such as Amazon S3, Google Cloud Storage or Azure Blob Storage, which is built for durable, cheap, very large objects. The metadata (owner, name, size, status, storage key) goes into a normal database. Clients upload and download directly to storage with short-lived signed URLs, and background workers process files after they arrive.
What you will be able to do
- Design an upload path where clients send bytes straight to object storage with presigned URLs.
- Make large uploads resumable with multipart upload, and size the parts.
- Keep the database record and the stored object in step, and clean up uploads that never finish.
- Serve downloads through a CDN with range requests and signed URLs.
- Verify integrity with checksums, and deduplicate with content hashes.
Key concepts
- Object storage
- Metadata vs bytes
- Presigned URLs
- Multipart upload
- Resumable upload
- Upload state machine
- Post-upload processing
- Range requests and CDNs
- Checksums and content addressing
Interview questions
- Design the upload path for a video-sharing site where files can be several gigabytes.
- How do presigned URLs work, and what are their risks?
- How do you make a 20 GB upload survive a flaky connection?
- Why not store files in the database?
- How would you design file sync for a product like Dropbox?
- How do you serve large files to many users efficiently?
- How do you stop users from uploading malicious or oversized files?
- Estimate storage and bandwidth for a photo app with 10 million daily uploads.
Sign up free to read the full topic
More System Design topics
Start free on Cariara · Coding problems · System design studio