Multi-Modal Data Store

Store and search images, text, audio and video together

Design one store for images, text, audio and video that keeps the files cheaply, describes them with shared metadata, and lets a text query find matching images or clips with filters applied.

Overview

A multi-modal data store keeps every kind of asset (text, images, audio, video) behind one API. Each asset has three parts: the raw bytes, metadata (owner, license, tags, dates) and derived data made by models: captions, transcripts, keyframes and an embedding, a list of numbers that places the asset in a space where similar things sit close together.

The reason to build one store instead of one per team is search across types. If images and text share an embedding space, the text 'a red bicycle on a beach' can find photos and video clips directly. Models such as CLIP (text and images) and ImageBind (six modalities) were trained for exactly this.

Two things make the problem hard. Sizes span six orders of magnitude, so uploads, storage classes and reads must handle a 20 KB document and a 2 GB video with one API. And a query like 'photos like this one, licensed CC-BY, from 2025' mixes an exact filter with an approximate vector search, which are indexed in different ways.

Questions to ask first

Which modalities? — Text, image, audio and video
How many assets and how big? — 1 billion assets, about 7.6 PB (mix below)
Who searches? — Internal ML teams and apps built on the store; about 20 million searches a day
Must search find text-to-image matches? — Yes, through one shared embedding model
How fresh must search be? — A new asset is searchable within minutes; bytes are readable at once
Are old assets read again? — Rarely, but unpredictably, so tiering must not make them unreachable by surprise

What you will be able to do

Size the bytes, the vectors and the search load from a stated asset mix.
Explain how CLIP-style models make a text query find images, and why one index must hold one model's vectors.
Compare post-filtering, pre-filtering and filterable HNSW, and show why post-filtering can return too few results.
Design resumable uploads with content-addressed deduplication that is safe under concurrent uploads.
Choose between age-based and access-based storage tiers, and serve large videos with range requests.

Key concepts

Content addressing
Multipart upload
Shared embedding space
Filtered vector search
Payload index
Storage tiers
Range requests
Derived artifacts
Idempotent processing

Functional requirements

Upload an asset of any size through one API; large files resume after a network failure
Store identical bytes once, even when two teams upload the same file
Run processing per modality: thumbnails and captions for images, transcripts for audio and video, keyframes for video
Embed every asset (or each keyframe of a video) into one shared space
Search by text, image or audio, restricted by metadata filters (modality, license, tags, dates)
Read or stream any asset, including byte ranges of large files
Keep derived data versioned by the model that made it

Non-functional requirements

Search answers in under 500 ms at p99 (a target we chose)
Bytes are durable once the upload call returns; derived data may lag by minutes
A new image is searchable within 5 minutes; a long video within an hour (targets we chose)
Storage cost falls as assets age, without making unpredictable reads fail
Processing is idempotent: a retried job never creates a second caption or vector

Interview questions

How does a text query find images?
Why is filtered vector search hard, and what do you do about it?
How do you make uploads of 50 GB files reliable?
How does deduplication work, and how do you delete safely?
How do you store and search video?
How do you cut storage cost without breaking reads?
What happens when you switch to a better embedding model?