Conversation Memory & Context Windows

An LLM API call is stateless. The model does not remember your previous message on its own, every single call is a fresh, independent request with no memory of anything that came before it. Whatever "memory" a chatbot appears to have is an illusion your application builds by resending the relevant conversation history as part of every new prompt. Getting this right is the entire subject of this chapter, and getting it wrong is one of the most common production bugs in real GenAI systems.

A model's context window has to hold three things at once, and they all compete for the same fixed token budget: the system prompt (instructions that set the model's behavior, present on every call), the conversation history (everything said so far that you choose to resend), and room reserved for the model's own response. Grow the history without bound, and it does not just get expensive, per the cost math from two chapters ago, it eventually leaves no room for a full response at all, which is exactly the "answers cut off mid-sentence" failure this whole module opened with.

• -Sliding window: keep only the most recent N turns verbatim, and simply drop anything older. Cheap and simple, but genuinely loses information, ask about something from ten turns ago and the model has no way to know it happened.
• -Rolling summarization: periodically compress older turns into a short summary (itself generated by an LLM call) and keep only the summary plus the most recent few turns verbatim. More expensive (it costs an extra model call to produce the summary) and lossier in a different way (details get compressed away), but it preserves a working memory of the whole conversation instead of a hard cliff.
• -A common hybrid, and the one most production systems actually reach for: keep recent turns verbatim for fidelity, and maintain a running summary of everything older, exactly the strategy the interactive demo above just walked through.

Conversation memory, as covered in this chapter, is short-term and scoped to a single session: it disappears the moment the conversation ends. A different kind of memory, long-term memory that persists facts about a user or task across entirely separate sessions ("remember that I prefer metric units," carried into a conversation next week), is a distinct problem with a distinct set of tools, and it gets its own full treatment in the agents tier ahead, once you have tools and retrieval available to build it properly.