Semantic Cache for LLM Responses

Reuse answers for questions that mean the same thing

Design a cache that sits in front of a large language model and returns a stored answer when a new question means the same as one already answered, without ever serving one customer's answer to another.

Overview

A semantic cache stores answers from a large language model (LLM) and reuses one when a new question means the same thing as an old one. 'What is your refund policy?' and 'How do refunds work?' are different strings, so a normal cache keyed on the exact text misses. A semantic cache turns each question into an embedding (a list of numbers where similar meanings land close together), finds the nearest stored question, and returns its answer if the two are close enough.

The payoff is fewer model calls: a hit skips the slowest and most expensive step. The risk is a wrong hit: 'capital of France' and 'capital of Germany' are close in wording, and serving one answer for the other is worse than no cache at all. A 2025 paper on semantic caching (vCache) reports that one fixed similarity threshold for all requests gives no correctness guarantee and leads to unexpected error rates, which is why this design treats the threshold as something you measure, not a constant you pick.

Questions to ask first

What traffic? — 20 million LLM requests a day across 500 business customers (tenants)
What kind of questions? — Support and FAQ assistants, where many users ask the same things
Are answers personal? — Some are; those skip the cache or are scoped to the user
How wrong may a hit be? — At most about 1 wrong hit in 100, checked by sampling
Can tenants share answers? — Never
Is the cache on the critical path? — It must not be: if it fails, requests go straight to the model

What you will be able to do

Explain the lookup and store paths and why a miss must never be slower than no cache by much.
Choose a similarity threshold from labeled pairs and say what it costs in hit rate.
Define the cache scope so a hit can never cross a tenant, model, prompt version or user.
Invalidate answers by time, by version bump and by source document.
Compare a semantic cache with exact caching and with provider prompt caching.

Key concepts

Embedding
Cosine similarity
Similarity threshold
Approximate nearest neighbor
Cache scope
Tenant isolation
TTL and versioned invalidation
Single-flight misses
Wrong-hit rate

Functional requirements

Before calling the model, look up the nearest stored question for this tenant and scope
Return the stored answer when similarity clears the tenant's threshold; otherwise call the model
Store new question and answer pairs with a time to live (TTL) and the documents they used
Let each tenant set its threshold and TTL, and opt routes out of caching
Invalidate by model or prompt version, by source document, or all at once
Report hit rate, latency saved and a sampled wrong-hit rate per tenant

Non-functional requirements

A lookup adds under 30 ms at p99 on a miss (a target we chose), since every miss pays it
No answer ever crosses tenants, under any bug in a single layer
Wrong-hit rate under about 1% on sampled hits (a target we chose)
Cache failure means a normal model call, never an error to the user
A changed document or model stops matching old answers within minutes

Interview questions

How do you pick the similarity threshold?
How do you make sure one tenant never sees another tenant's answers?
What should never be cached?
How do you invalidate a semantic cache?
How is this different from the prompt caching LLM providers offer?
What happens when many users ask the same new question at once?
How big does the index get, and how do you shrink it?