RAG System (Retrieval-Augmented Generation)

Question Answering over a Company's Documents

Design a system that answers questions from a company's own documents: it finds the right passages with keyword and vector search, hands them to a language model, and returns an answer with citations that respects who may read what.

Overview

A language model knows only what was in its training data. It does not know your company's refund policy, last week's incident report or the contract signed yesterday, and when asked, it may produce a confident guess. Retrieval-augmented generation (RAG) fixes this by looking up relevant passages at question time and giving them to the model with the question, so the answer is based on your documents and can cite them.

The name comes from Lewis et al. (2020), who paired a sequence-to-sequence model (the parametric memory) with a dense vector index of Wikipedia searched by a neural retriever (the non-parametric memory). Production systems today usually keep the model fixed and put the engineering into the retrieval side: how documents are split, how passages are found, who may see them, and how quality is measured.

In an interview, the model call is one box. The design is everything around it: ingestion that keeps the index fresh, retrieval that finds the right passage when the words differ and when they match exactly, permissions that never leak a document, and an evaluation loop that tells you whether a change helped.

Questions to ask first

What documents, how many? — Wiki pages, PDFs, tickets: 10 million documents across many companies (tenants)
Who asks, how often? — Employees; 2 million questions a day
Do permissions matter? — Yes: a user must never see a passage from a document they cannot open
How fresh? — An edited document is answerable within 10 minutes (a target we chose)
How fast? — First words within about 1.5 seconds, streamed (a target we chose)
Which model? — Any hosted model behind an interface; not our choice here

What you will be able to do

Size a RAG system: chunks, vector storage, question rate and token volume.
Chunk documents along their structure and keep each chunk understandable on its own.
Combine BM25 and vector search with reciprocal rank fusion, then rerank.
Enforce tenant and document permissions inside retrieval, and keep the index fresh.
Measure retrieval and answers separately, with a golden set and production traces.

Key concepts

Chunking
Embeddings
BM25
Hybrid retrieval
Reciprocal rank fusion
Reranking
Grounded generation
Citations
Permission filtering
Retrieval evaluation

Functional requirements

Connect sources and ingest documents (PDF, HTML, Markdown, office files) with their permissions
Re-index a document when it changes or is deleted
Answer a question with passages from documents the user may read, citing each one
Say so when the documents do not contain the answer
Stream the answer as it is generated
Collect feedback (helpful or not) on each answer

Non-functional requirements

Never return or use a passage from a document the user cannot open
First words within about 1.5 s at p95; full answers in a few seconds (targets we chose)
Edits and deletions reflected within 10 minutes
Every answer traceable: question, retrieved chunks, prompt version, model and answer are logged
Quality changes (chunking, retrieval, prompt, model) are measured before they ship

Interview questions

Why not just put all the documents in the model's context?
How do you choose a chunking strategy?
Why hybrid search, and how do you combine the results?
How do you stop the system from leaking documents between users?
How do you evaluate a RAG system?
What is HyDE, and when would you use it?
How do you keep answers fresh when documents change?
Where does the latency go, and how do you cut it?