Beyond Basic RAG: Better Queries, Chunks & Placement

A basic RAG pipeline splits documents into chunks, embeds them, retrieves the nearest few, and hands them to a model. It works, and then it plateaus. The earlier RAG chapters covered that pipeline and the evaluation that shows where it fails. This chapter is the toolbox for fixing those failures, matched to the failure each tool addresses.

A good research librarian does not just search for your exact words. They rephrase your question, check the index under several terms, note which book a snippet came from before photocopying it, and put the most important pages on top of the pile. Each of these is an upgrade to retrieval, and each solves a different problem.

Use the retrieval metrics from the evals module to find which stage fails, then pick the tool.

• -The question and the document use different words. Query rewriting and expansion generate alternative phrasings, or several sub-queries, and merge the results. HyDE asks a model to write a hypothetical answer first, then searches with that text, which often sits closer to real documents than a short question does.
• -A chunk loses its meaning when cut out of its document. "Revenue grew by 3%" does not say which company or year. Contextual retrieval adds a short, chunk-specific description to each chunk before indexing. Parent-child chunking retrieves a small chunk and returns the larger section around it.
• -Meaning matches but exact terms are missed, such as an error code. Hybrid search combines keyword matching and embeddings, as covered earlier.
• -The right chunk is retrieved but ranked low. A reranker rescores the shortlist with a stronger model.
• -Results ignore obvious constraints, like date or product line. Metadata filters narrow the search before or during retrieval.
• -Answers need facts spread across many documents or links between entities. Graph-based indexes record relationships, at a higher cost to build.

Anthropic's write-up on contextual retrieval gives concrete numbers. A model reads each document and writes a sentence or two situating each chunk, for example the company and the quarter, which is prepended to the chunk before it is embedded and before the keyword index is built. They measured how often none of the top 20 retrieved chunks contained the answer. The baseline missed 5.7% of questions. Contextual embeddings cut that to 3.7%, a 35% reduction. Adding contextual keyword search brought it to 2.9%, a 49% reduction, and adding a reranking step reached 1.9%, a 67% reduction. They estimate a one-time cost of about $1.02 per million document tokens, under their stated assumptions, with prompt caching.

The first part of the code below turns those reported rates into counts per thousand questions and a corpus-level cost. The second part is a toy that shows why placement matters, which the next section explains. Press Run, and read the first table from top to bottom.

The reported misses fall from about 57 in every 1,000 questions to about 19. The corpus-level cost is small: $51 for 50 million tokens, one time. These figures come from Anthropic's own test sets and prompts, so your results will differ, which is why you measure with your own labeled questions. The second table is a toy, with weights invented to look like the finding in the next section. Putting the most relevant chunk first scores 2.26. Putting the most relevant chunks at the two edges scores 2.56, and putting the best ones last scores 1.93.

Liu and colleagues studied how language models use information in long inputs. In their experiments, performance was often highest when the relevant passage was at the beginning or the end of the context, and dropped when it was in the middle, even for models built for long contexts. This is sometimes called the lost-in-the-middle effect. Models differ, and newer ones may do better, so measure your own, but the practical advice holds. Keep retrieved context short, put the strongest chunks at the start and the end, and do not pad the prompt with weak ones.

• -Long context is simple and avoids retrieval misses when everything fits. You pay for all those tokens on every call, latency grows, and the model may not use the middle well.
• -Retrieval sends only relevant text, so it costs less per query and scales to corpora far beyond any context window. It can miss, which is why retrieval quality must be measured.
• -A common pattern is both: retrieve a shortlist, then give the model enough surrounding text, in a prompt that is cached when it repeats.