Assembling Context Windows for Generation
You'll learn to
- -Explain why more retrieved chunks is not automatically better
- -Run the assembly steps: filter by relevance, remove near-duplicates, fit a budget, order for attention
- -Describe the "lost in the middle" effect and design around it
A code assistant retrieves ten relevant snippets for a query and stuffs every one into the prompt. The model gets confused by contradictory examples, two snippets showing different, incompatible ways to do the same thing, and writes buggy code. Retrieval did its job, arguably too well. Context assembly, the step that turns "here are ten relevant things" into "here is one well-organized prompt", is where this system failed.
An Analogy: A Briefing Memo for a Busy Executive
When you brief a busy executive, you do not hand over the whole filing cabinet. You pick the few pages that matter, remove repeats, keep the memo to a page, and put the key point at the top. The executive reads the start and the end most carefully. Assembling a prompt is writing that memo for a model.
More Retrieved Chunks Is Not Automatically Better
Recall the context window budget from the earlier tier: every token spent on retrieved context is a token not available for the answer. And beyond the budget, dumping in every chunk unfiltered risks near-duplicates, contradictory information and low-relevance filler, which dilute the useful context with noise the model must sift through.
The "Lost in the Middle" Effect
A well-documented effect in long-context LLM behavior: models attend most reliably to information near the beginning and end of a long context, and are measurably more likely to under-use or miss information in the middle. In practice, the single most relevant chunk should usually go first, or in some schemes last, not buried at position five of ten, purely because of where a model tends to look.
Deduplication
Overlapping chunks (recall the overlap setting from the chunking chapter) and near-duplicate documents routinely retrieve several chunks that say almost the same thing. Including all of them wastes token budget and adds no new information. It can even reinforce a wrong or outdated fact, if an old and a new document both retrieved for the same query disagree. A deduplication pass, which checks retrieved chunks for high similarity and drops near-copies, is a cheap, high-value step before assembly.
Parent-Doc Retrieval, Applied at Assembly Time
The parent-child chunking pattern from earlier in the tier pays off here. When assembling the context, expanding a retrieved small chunk to its full parent section, not just the narrow matched piece, often gives the model enough surrounding context to use the information properly, at the cost of some extra tokens per chunk.
The demo runs the assembly steps on six reranked chunks, with a budget of 30 words. One chunk is a near-copy of the first, and one is irrelevant filler. Watch the list shrink at each step, and where the best chunks end up.
A practical checklist: rank chunks by actual relevance (after reranking, not raw retrieval order), remove near-identical content, respect the token budget by cutting the list instead of cutting individual chunks mid-sentence, and put the highest-relevance chunk where the model is most likely to attend.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!