Multi-Modal AI Search Engine

Overview

A multi-modal search engine lets people search the way they think: type 'mid-century armchair', upload a photo of a chair they saw in a café, speak a query, or upload the photo and add 'but in green'. Behind all four is one idea: turn both the query and every indexed image into a vector (a list of numbers) such that similar meaning lands close together, whatever form it came in.

Models trained like CLIP make this possible. CLIP was trained on 400 million image and caption pairs to predict which caption goes with which image, and the result is a text encoder and an image encoder whose vectors live in the same space. A query is then a nearest-neighbor search among a billion vectors, which is the core engineering problem: exact search is far too slow, so the index is approximate, sharded and compressed.

Questions to ask first

• What is being searched? — 1 billion images, most with a title or product listing
• Which query types? — Text, photo, voice, and photo plus words
• What comes back? — A ranked list of images with their pages or product listings
• How fast? — p95 under 500 ms for photo queries, 300 ms for text (targets we chose)
• How fresh? — New and changed images searchable within an hour
• Filters? — Category, color, price, and safe search on by default
• Traffic? — 100 million searches a day, 30% of them with a photo

What you will be able to do

• Explain how a shared embedding space lets a photo find text and text find photos, and where it fails.
• Size a billion-vector index in memory and choose between HNSW and compressed IVF-PQ.
• Shard the index, merge results with scatter-gather, and fuse vector and keyword results.
• Handle a photo plus words query with filters instead of averaging vectors.
• Keep the index fresh, filter unsafe images, and upgrade the embedding model without downtime.

Key concepts

• Shared embedding space (CLIP)
• Approximate nearest neighbors
• HNSW
• IVF and product quantization
• Scatter-gather sharding
• Hybrid search and RRF
• Two-stage ranking
• Composed queries
• Perceptual hashing
• Embedding versioning

Functional requirements

• Search by text, by photo (upload or URL), by voice, or by a photo plus words
• Return a ranked page of results with image, title and link
• Filter by category, color, price range; safe search on by default
• Find visually similar images to any indexed image
• Index new images and re-index changed ones within an hour
• Log queries, results and clicks for ranking and evaluation

Non-functional requirements

• p95 latency under 300 ms for text queries and 500 ms for photo queries (targets we chose)
• About 3,500 searches a second at peak, with room to grow by adding replicas
• Degrade instead of failing: if the vector tier is slow, answer from the keyword index
• Recall of the approximate search measured continuously against exact search on a sample
• Unsafe images never shown with safe search on

Interview questions

• How can a photo find text results and text find photos?
• How do you search a billion vectors in tens of milliseconds?
• HNSW or IVF-PQ?
• How do you handle "this sofa, but in blue leather"?
• How do you combine keyword and vector results?
• How do you keep the index fresh and upgrade the model?
• How do you keep unsafe images out of results?
• What happens when a shard is slow or down?

More System Design topics

• ML Experiment Tracking System
• ML Feature Store
• ML Training Data Lake
• Model Registry and Artifact Store
• Monitoring and Observability
• Monolith vs Microservices
• Multi-Modal Data Store
• Multi-Region Active-Active Architecture
• Multi-Step Processes
• Multi-Tenancy Architecture
• Multi-Tenant SaaS Platform
• Multi-Version Concurrency Control