Skip to content
GenAI Learn/Building the RAG Pipeline
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Assembling Context Windows for Generation

4 min read

You'll learn to

  • -Explain why more retrieved chunks is not automatically better
  • -Run the assembly steps: filter by relevance, remove near-duplicates, fit a budget, order for attention
  • -Describe the "lost in the middle" effect and design around it

A code assistant retrieves ten relevant snippets for a query and stuffs every one into the prompt. The model gets confused by contradictory examples, two snippets showing different, incompatible ways to do the same thing, and writes buggy code. Retrieval did its job, arguably too well. Context assembly, the step that turns "here are ten relevant things" into "here is one well-organized prompt", is where this system failed.

An Analogy: A Briefing Memo for a Busy Executive

When you brief a busy executive, you do not hand over the whole filing cabinet. You pick the few pages that matter, remove repeats, keep the memo to a page, and put the key point at the top. The executive reads the start and the end most carefully. Assembling a prompt is writing that memo for a model.

More Retrieved Chunks Is Not Automatically Better

Recall the context window budget from the earlier tier: every token spent on retrieved context is a token not available for the answer. And beyond the budget, dumping in every chunk unfiltered risks near-duplicates, contradictory information and low-relevance filler, which dilute the useful context with noise the model must sift through.

The "Lost in the Middle" Effect

A well-documented effect in long-context LLM behavior: models attend most reliably to information near the beginning and end of a long context, and are measurably more likely to under-use or miss information in the middle. In practice, the single most relevant chunk should usually go first, or in some schemes last, not buried at position five of ten, purely because of where a model tends to look.

Deduplication

Overlapping chunks (recall the overlap setting from the chunking chapter) and near-duplicate documents routinely retrieve several chunks that say almost the same thing. Including all of them wastes token budget and adds no new information. It can even reinforce a wrong or outdated fact, if an old and a new document both retrieved for the same query disagree. A deduplication pass, which checks retrieved chunks for high similarity and drops near-copies, is a cheap, high-value step before assembly.

Parent-Doc Retrieval, Applied at Assembly Time

The parent-child chunking pattern from earlier in the tier pays off here. When assembling the context, expanding a retrieved small chunk to its full parent section, not just the narrow matched piece, often gives the model enough surrounding context to use the information properly, at the cost of some extra tokens per chunk.

The demo runs the assembly steps on six reranked chunks, with a budget of 30 words. One chunk is a near-copy of the first, and one is irrelevant filler. Watch the list shrink at each step, and where the best chunks end up.

Press Run: six chunks become a short, deduplicated, budgeted prompt with the best at the edges

A practical checklist: rank chunks by actual relevance (after reranking, not raw retrieval order), remove near-identical content, respect the token budget by cutting the list instead of cutting individual chunks mid-sentence, and put the highest-relevance chunk where the model is most likely to attend.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!