Text Summarization System

Summaries of Documents, Threads and Meetings

Design a service that turns documents, email threads, chat channels and meeting transcripts into short summaries in the format the reader asked for, including documents far longer than a model can read in one call.

Overview

A summarization service takes text a person does not have time to read (a 60-page report, a 200-message email thread, a one-hour meeting transcript) and returns the part they need, in the shape they asked for: three bullets, one paragraph, or a list of action items with owners.

The model call is the easy part. The design work is around it. A large language model (LLM) reads a limited number of tokens (pieces of words; OpenAI's rule of thumb for English is about 4 characters or three quarters of a word per token) in one call, its context window. Long inputs must be split and the pieces combined without losing or inventing facts. The same popular document is summarized for many people, so caching matters for cost. And a summary is read instead of the source, so an invented number does real harm.

Questions to ask first

What inputs? — Documents, email threads, chat channels, meeting transcripts (already text)
How long can an input be? — Up to about 500 pages; most are under 10
What formats? — Bullets, paragraph, action items; short, medium or long
How fast? — Short inputs stream the first words within 2 seconds; long ones may take a minute
Who may see a summary? — Only people who may read the source
How big is the model's window? — We assume 100,000 tokens usable per call; we cap one-call summaries lower, at 20,000 tokens, and test that cap on real documents

What you will be able to do

Size a summarization service in tokens, the unit that drives both latency and cost.
Summarize documents far longer than the context window with map-reduce, and say what it loses.
Cache and deduplicate summaries without leaking content across users or companies.
Check a summary against its source, and say what automatic checks can and cannot catch.

Key concepts

Context window
Tokens
Map-reduce summarization
Chunk overlap
Faithfulness checks
Content-hash cache
Single-flight jobs
Streaming (SSE)
Structured output
Tenant isolation

Functional requirements

Summarize a document, a thread or a transcript in a chosen format and length
Stream the summary for short inputs; run long inputs as jobs with progress
Pull out action items with owner and due date when the source states them
Reuse an existing summary when the same source, format and length were summarized before
Let users rate or edit a summary; keep that feedback for evaluation
Show a summary only to users who can read its source

Non-functional requirements

First streamed words within 2 seconds for inputs under 10 pages (a target we chose)
A 500-page document done within about 1 minute (a target we chose)
No names, numbers or dates in a summary that the source does not contain; suspected ones are flagged
No summary or cached text ever crosses a tenant boundary
The service degrades to a clear error or a queued job, never a truncated summary presented as complete

Interview questions

How do you summarize a document longer than the context window?
How do you know the summary is correct?
How do you cache summaries safely?
How is a thread or meeting different from a document?
Where does the cost go, and how do you cut it?
What happens when the model provider is slow or down?