Object Storage (S3)

Blob Storage Service

Design an object store like Amazon S3: a flat namespace of buckets and keys, strongly consistent metadata, bytes spread across disks and zones with erasure coding, and background repair that keeps data safe for years.

Overview

An object store keeps objects: a blob of bytes plus a little metadata, stored under a key inside a bucket. There are no directories and no in-place edits; you put a whole object, get it (or a byte range of it), and delete it. That narrow interface is what lets the system grow to trillions of objects: there is no directory tree to keep consistent and no file locking.

The interview has two halves. The metadata side answers 'does bucket/key exist, which version is current, where are its bytes?' It is small per object, must be strongly consistent, and serves every request. The data side stores the bytes: petabytes on cheap disks that fail every day, spread so that losing a disk, a rack or a whole building loses nothing. Amazon documents S3 Standard as designed for 99.999999999% durability and 99.99% availability over a year, storing objects across at least three Availability Zones. Its internals are not public, so this page names two systems that published theirs: Dropbox's Magic Pocket and Ceph's CRUSH placement algorithm.

The client-facing patterns (presigned URLs, multipart uploads from browsers, CDNs, range requests) are in the Handling Large Blobs topic. This page designs the store underneath.

Questions to ask first

How much data? — About 100 billion objects, 1 MB on average, 100 PB in total
Object sizes? — From a few bytes to tens of terabytes (multipart)
Traffic? — 500,000 reads and 50,000 writes a second
Consistency? — A read or list after a successful write sees it, as S3 documents
Durability? — Survive the loss of a whole zone with no data loss
Mutations? — Whole-object replace only; optional versioning

What you will be able to do

Separate metadata from data and say what each store must guarantee.
Place an object's pieces across zones without a central lookup for every byte.
Compare replication and erasure coding on overhead, durability and repair cost, with numbers.
Design a write path that gives read-after-write consistency and conditional writes.
Explain listing, repair, scrubbing, deletes and storage tiers.

Key concepts

Metadata vs data
Flat namespace
Placement without a directory
Failure domains
Erasure coding
Repair window
Read-after-write consistency
Conditional writes
Prefix listing
Scrubbing and garbage collection

Functional requirements

Create buckets; put, get (whole or a byte range), head and delete objects by key
Upload large objects in parts and assemble them
List keys by prefix with a delimiter, one page at a time
Optional versioning: keep every version; delete adds a marker
Conditional writes: create only if absent, replace only if the ETag matches
Lifecycle rules: move objects to colder tiers or expire them

Non-functional requirements

Survive the loss of any one zone with no data loss and no read outage (a target we chose)
Read-after-write consistency for puts, deletes and lists
Scale out by adding disks and prefixes, with no fixed limit on objects per bucket
Storage overhead well under 3× (a target we chose)
First byte in about 100 to 200 ms for small objects (the range S3 documents)

Interview questions

Why separate metadata from data?
How do you get read-after-write consistency?
Replication or erasure coding? Show the trade-off.
How would you reach eleven nines?
How does listing work on a flat namespace?
How do you prevent two writers from clobbering each other?
A disk dies. Walk through what happens.
How do you store billions of tiny objects efficiently?