Context Window Management for Persistent Agents

Overview

Overview

Production agents that run for days or weeks hit a hard physical limit: the LLM context window. Even large windows (GPT-4o offers 128K tokens) fill up, because a multi-day conversation with tool round-trips, retrieved documents and verbose responses can add tens of thousands of tokens per hour, and the raw history passes the limit within one business day.

> [!WARNING] Context overflow is a user experience failure, not just a technical error. An agent that forgets a constraint the user stated two hours ago is not useful as a persistent assistant.

Four strategies

• Sliding window — Keep only the last N messages — Simple, built-in trim_messages() — Early context is permanently lost
• Summarization node — Compress old messages into a running summary past a threshold — Keeps key decisions in few tokens — Detail outside the summary is discarded
• Vector DB offloading — Embed every exchange, retrieve the top-k relevant ones per turn — No information is lost — Retrieval depends on semantic similarity
• Hybrid — Recent verbatim + running summary + vector retrieval — The production pattern — Needs both a summarizer and a vector DB

Strategy 1: sliding window truncation

Keep only the last N messages in the prompt with the trim_messages() utility:

> [!WARNING] A day-one constraint such as "never recommend solutions that require vendor lock-in" drops out of the window, and the model behaves as if it was never stated.

Strategy 2: summarization node

A conditional edge routes to summarize_node when the message or token count passes a threshold. The summary becomes a synthetic message at the top of the context, and the raw messages it replaced are discarded.

• Prompt: "Summarize the conversation, preserving all key decisions, facts, and constraints stated by the user."
• State update: replaces old messages with the summary plus recent messages.
• Incremental: the summary lives in a running_summary field, and each call extends it rather than starting over.

Strategy 3: vector DB offloading (episodic memory)

Every message exchange is treated as a document to embed and store.

• 1 — Offload old exchanges to a vector store (pgvector, Pinecone, Weaviate, Chroma)
• 2 — At the start of each turn, embed the current user message
• 3 — Retrieve the top-k most semantically relevant past exchanges

Strategy 4: hybrid architecture (production pattern)

• Recent — Last 15-20 exchanges — LangGraph messages state — Verbatim injection
• Medium-term — Exchanges from the last few hours — running_summary field in state — Always included
• Long-term — Exchanges from prior sessions — Vector DB (pgvector) — Semantic similarity retrieval

The LLM prompt structure:

The LangGraph Store API

LangGraph 0.3+ adds a Store for cross-thread, cross-session memory. Checkpointers scope state to a thread_id; the Store is a key-value namespace that persists independently of any graph run.

• InMemoryStore: the development option.
• PostgresStore: the production option, a PostgreSQL table indexed by namespace and key.

> [!TIP] A node that learns "user prefers concise answers" writes {"preference": "concise"} under the user's namespace, and every later graph run for that user reads it, whatever the thread_id.

Memory type taxonomy

• Semantic — Facts and propositions, changing slowly — "the user works in financial services", "the project uses PostgreSQL 15" — LangGraph Store or a user profile table
• Episodic — Event sequences — "on June 10 we decided on event sourcing" — Vector DB with timestamp metadata
• Procedural — Workflow knowledge — "to deploy this project, run X then Y then Z" — System prompt or a retrieved knowledge base

> [!IMPORTANT] Mixing all three into one undifferentiated message list is the architectural mistake that makes context management fail at scale.

Interview questions

• Explain how you manage the context window in a persistent, multi-turn agent that might have a conversation spanning several days.
• Walk me through the LangGraph code for a summarization node that fires when the conversation exceeds 4000 tokens.
• How do you ensure the agent remembers a key fact the user mentioned 3 days ago without storing the entire raw conversation?

More Agentic AI topics

• Async Long-Running Tool Calls
• Multi-Agent Systems with LangGraph