Vector Databases Deep Dive

How vector search works and how to run it: embeddings and distance, why exact search does not scale, HNSW, IVF and quantization, filtering, hybrid search with rank fusion, retrieval for LLMs, and when pgvector is enough.

Overview

A vector database stores embeddings, lists of numbers that a machine-learning model produces from text, images or other data, and finds the stored vectors closest to a query vector. Items with similar meaning get nearby vectors, so 'closest vectors' means 'most similar items', even when they share no words.

The hard part is speed. Finding the exact nearest neighbors means comparing the query with every stored vector, which is too slow at millions of vectors. Vector databases instead use approximate nearest neighbor (ANN) indexes that look at a small part of the data and usually, but not always, find the true nearest neighbors. How often they do is called recall.

Vector search can be a feature of a database you already run, such as the pgvector extension for PostgreSQL, or a dedicated system such as Milvus, Qdrant, Weaviate or the managed Pinecone service. The ideas are the same everywhere: an index, a distance, a recall target, filters, and the memory it all takes.

What you will be able to do

  • Explain embeddings and pick a distance function.
  • Explain how HNSW and IVF indexes find neighbors, and tune them for recall or speed.
  • Estimate the memory an index needs, and use quantization to cut it.
  • Combine vector search with filters and with keyword search.
  • Design the retrieval step of an LLM application, and choose between pgvector and a dedicated system.

Key concepts

  • Embeddings
  • Cosine, dot product and L2
  • Approximate nearest neighbor
  • HNSW
  • IVF
  • Quantization
  • Recall
  • Filtered search
  • Hybrid search and RRF
  • Retrieval-augmented generation

Interview questions

  • How does HNSW work, and how do you tune it?
  • Estimate the memory for 100 million 768-dimension vectors with HNSW.
  • How do you combine vector search with metadata filters?
  • What is hybrid search and why use it?
  • Design the retrieval layer for a company knowledge assistant.
  • When is pgvector enough, and when would you move to a dedicated vector database?
  • What is recall@k, and how do you measure it?
  • Why might cosine, dot product and Euclidean distance give the same results?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio