Embedding Generation Pipeline

Embed Billions of Chunks for Semantic Search

Design a pipeline that turns a billion text chunks into vectors, keeps the vector index fresh as documents change, embeds search queries in real time, and swaps in a new embedding model without downtime.

Overview

Semantic search and retrieval-augmented generation (RAG) look up text by meaning. An embedding model turns each piece of text into a vector (a list of a few hundred to a few thousand numbers) so that texts with similar meaning get nearby vectors. A search embeds the query with the same model and asks a vector index for the nearest vectors.

The model call is the easy part. The system around it has three hard jobs: embed a very large corpus without wasting GPU time; keep the index in step as documents are added, edited and deleted; and change the model, which makes every stored vector incompatible, without taking search down.

Questions to ask first

How big is the corpus? — 1 billion chunks of about 256 tokens each
Our own model or an API? — A self-hosted encoder with 768 dimensions; an API is compared in the estimate
How fresh must search be? — A changed document is searchable within 5 minutes (our target)
How many searches? — 100 million a day, each needing a query embedding
How often does the model change? — A few times a year; each change re-embeds everything
Are deletes important? — Yes: a deleted document must stop appearing in results

What you will be able to do

Size the job: GPU hours, storage for a billion vectors, and the API alternative's cost.
Chunk documents and use a content hash so unchanged text is never embedded twice.
Keep GPUs busy with batching, length bucketing and resumable shards.
Keep the index fresh under updates and deletes, and explain what HNSW and IVF-PQ cost.
Migrate to a new embedding model with a parallel index and an atomic switch.

Key concepts

Chunking
Content hash
Batch inference
Padding and length bucketing
L2 normalization
HNSW
Product quantization
Upserts and tombstones
Model versioning
Alias switch

Functional requirements

Split documents into chunks and embed each chunk with the current model
Skip chunks whose text and model are unchanged (content hash)
Backfill the whole corpus with a new model while the old index keeps serving
Embed new and changed chunks, and remove deleted ones, within minutes
Embed search queries in real time with the same model as the index being searched
Record the model version on every vector; never mix versions in one index

Non-functional requirements

A changed document is searchable within 5 minutes (a target we chose)
Query embedding adds under 30 ms at p99 (our target)
A backfill survives worker failures and resumes without redoing finished shards
GPUs stay busy during backfill: large batches, little padding, no waiting on input
Model switches and rollbacks happen with no failed queries

Interview questions

How long does embedding a billion chunks take, and what does it cost?
How do you avoid re-embedding the whole corpus every night?
How do you migrate to a new embedding model with no downtime?
How do you keep GPUs busy during the backfill?
How are deletes handled in a vector index?
HNSW or IVF-PQ for a billion vectors?
How do you know the embeddings are still good?