Skip to content
GenAI Learn/Tokens, Embeddings & Memory
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Conversation Memory & Context Windows

5 min read

You'll learn to

  • -Explain why an LLM call is stateless and how a chatbot's memory is built by resending history
  • -Budget a context window across system prompt, history and response, and find when history stops fitting
  • -Compare a sliding window with rolling summarization, and name what each forgets

An LLM API call is stateless. The model does not remember your previous message on its own. Each call is a fresh, independent request with no memory of what came before. Whatever "memory" a chatbot appears to have is an illusion your application builds by resending the relevant conversation history with every new prompt. Getting this right is the whole subject of this chapter, and getting it wrong is one of the most common production bugs in real GenAI systems.

An Analogy: A Helper Who Reads Your Notepad Each Time

Imagine a very capable helper with no memory at all. Every time you speak to them, you must hand over a notepad with everything said so far. If the notepad gets too long to read in one sitting, you must decide what to cut or condense. The notepad is the conversation history, and how you keep it short is the topic of this chapter.

The Budget: System Prompt + History + Response, All Competing

A model's context window must hold three things at once, all competing for the same fixed budget of tokens. The system prompt is the instructions that set the model's behavior, present on every call. The conversation history is everything said so far that you choose to resend. And there must be room reserved for the model's own response. If history grows without bound, it does not only get expensive, as the cost math from two chapters back showed. It eventually leaves no room for a full response, which is the "answers cut off mid-sentence" failure this whole module opened with.

Try the interactive budget below first. Add turns and watch the bar fill, then see what happens when the history leaves too little room for the answer.

A Conversation Filling Its Context Window

A 2000-token toy window. Watch history grow until summarization kicks in.

0threshold 12002000 (window)
no history yet

A fresh conversation. 150 tokens are already reserved for the system prompt, and 300 are held back for the next response, leaving 1550 for history.

step 0 / 7

Two Strategies for Keeping History Bounded

  • -Sliding window: keep only the most recent N turns verbatim, and drop anything older. Cheap and simple, but it loses information. Ask about something from ten turns ago and the model has no way to know it happened.
  • -Rolling summarization: periodically compress older turns into a short summary (itself produced by an LLM call), and keep the summary plus the most recent few turns verbatim. It costs an extra model call and loses detail in a different way, because specifics get compressed away, but it keeps a working memory of the whole conversation instead of a hard cliff.
  • -A common hybrid, which most production systems reach for: keep recent turns verbatim for fidelity, and maintain a running summary of everything older.

The demo works through the numbers. The window is 1,000 tokens, the system prompt uses 200 and 300 are reserved for the answer, so 500 are left for history. Each turn is about 80 tokens. See at which turn full history stops fitting, and how much each strategy uses.

Press Run: full history stops fitting at turn 7, and both strategies stay well inside the budget

This is exactly what the GenAI Lab's ConversationMemory summarize_after setting controls: the turn at which the pipeline switches from "just resend everything" to "start compressing the older turns". Set it too high and the window fills before summarizing begins. Set it too low and you waste model calls summarizing conversations that were never going to grow long.

A Second, Separate Kind of Memory

Conversation memory, as covered here, is short-term and belongs to one session: it disappears when the conversation ends. A different kind, long-term memory that keeps facts about a user or task across separate sessions ("remember that I prefer metric units", used again next week), is a separate problem with its own tools. It gets its own treatment in the agents tier ahead, once tools and retrieval are available to build it properly.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Put tokenization, embeddings, and memory into practice: build the Token Counter, Meaning Machine, and Memory Vault levels in the GenAI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!