Chunking Strategies That Actually Work

A RAG system retrieves relevant-looking chunks, but the answers keep coming out wrong. The reason: critical context is being split across chunk boundaries. A drug name mentioned in one sentence gets separated from its dangerous interaction described in the very next sentence, and whichever chunk gets retrieved contains only half the fact. This is one of the most common, hardest-to-notice bugs in a RAG system, because retrieval looks like it is working, the chunks it returns really are topically relevant, they are just incomplete.

The simplest strategy splits a document into fixed-length pieces (by character count or token count), optionally with some overlap between consecutive chunks so content near a boundary appears in more than one chunk. It is trivial to implement and fast to run over millions of documents, but it is also blind to sentence and paragraph structure, a boundary can and will land in the middle of exactly the wrong sentence.

At a chunk size of 90 characters, the critical fact splits across two separate chunks, exactly the failure mode above. At 150 characters, both sentences happen to fit together in the same chunk, and the fact survives intact. Notice this was not fixed by adding overlap, it was fixed by the boundary simply falling somewhere else. Overlap helps only when it is generous enough to actually bridge the specific gap between two related facts, which is not guaranteed just because some overlap exists at all.

Instead of cutting at a fixed character count, semantic chunking looks for natural breakpoints: embed consecutive sentences, and split where the similarity between adjacent sentences drops noticeably, the point where the topic actually shifts. This produces chunks that respect the document's real structure, at the cost of an extra embedding pass during ingestion and a less predictable chunk size (some topics are naturally covered in two sentences, others in twenty).

A genuinely useful middle ground: index small, precise chunks for search (a single paragraph, easy to match tightly against a specific query), but when one is retrieved, hand the model its larger parent chunk (the full section or document it came from) for actual generation. Small chunks make retrieval precise; large context makes generation well-informed. Keeping them as two different granularities, linked by a parent-child relationship in the metadata, gets both properties instead of trading one for the other.