Assembling Context Windows for Generation

A code assistant retrieves ten relevant snippets for a query and stuffs every single one into the prompt. The result: the model gets confused by contradictory examples (two snippets showing different, incompatible ways to do the same thing) and generates buggy code. Retrieval did its job, arguably too well, and context assembly, the step of turning "here are ten relevant things" into "here is one well-organized prompt," is where this system actually failed.

Recall the context window budget math from Tier 7: every token spent on retrieved context is a token not available for the response, and beyond the pure budget concern, dumping in every retrieved chunk unfiltered risks including near-duplicates, contradictory information, and low-relevance filler that dilutes the genuinely useful context with noise the model has to sift through.

A well-documented effect in long-context LLM behavior: models tend to attend most reliably to information near the beginning and the end of a long context, and are measurably more likely to under-use or miss information buried in the middle. This has a direct, practical implication for context assembly: the single most relevant retrieved chunk should generally be placed first (or, in some ordering schemes, last), not buried in position five of ten, purely because of where a model tends to actually look.

Overlapping chunks (recall the overlap setting from the chunking chapter) and near-duplicate documents in a corpus routinely retrieve multiple chunks that say almost the same thing. Including all of them wastes token budget without adding new information, and can even reinforce a wrong or outdated version of a fact if an old and a new document both retrieved for the same query happen to disagree. A dedication pass, checking for high similarity between retrieved chunks and dropping near-duplicates, is a cheap, high-value step before assembly.

The parent-child chunking pattern from earlier in this tier pays off directly here: when assembling the final context, expanding a retrieved small chunk out to its full parent section (rather than just the narrow, precisely-matched piece) often gives the model enough surrounding context to actually use the information correctly, at the cost of some additional tokens spent per chunk included.