Reranking for Precision

A medical RAG system retrieves ten chunks for every query, but the actually relevant one is frequently sitting at position five through eight, not position one. For a doctor scanning results under time pressure, or worse, for an LLM given all ten chunks with no signal about which one actually matters most, that buried ranking is a real, sometimes urgent, problem.

The embedding models used for the initial retrieval pass (covered across the last two chapters) are bi-encoders: the query and every document are embedded completely independently, with no interaction between them, and then compared by a simple, fast operation like cosine similarity. This independence is exactly what makes vector search fast enough to search millions of documents: every document's embedding is precomputed once, ahead of any query, and a new query only has to be embedded and compared once each. The cost of that speed is precision: the model never actually reads the query and a specific document together, so it can miss nuanced relevance that only becomes clear when the two are considered jointly.

A cross-encoder takes the query and a single candidate document together, as one combined input, and outputs a direct relevance score for that specific pair. Because it reads both texts jointly, in the same forward pass, it captures interactions a bi-encoder's independent embeddings structurally cannot, but that also means it has to be run once per candidate document, no precomputation possible, which makes it far too slow to run against an entire corpus of millions of documents.

This is exactly why reranking is a second stage layered on top of retrieval, not a replacement for it. Vector search (a bi-encoder, cheap and precomputable) narrows millions of documents down to a manageable shortlist, typically the top 20 to 100 candidates. A cross-encoder then reranks only that small shortlist, applying its more expensive but far more accurate scoring where it can actually afford to. Running a cross-encoder against the whole corpus for every query would be prohibitively slow; running only a cheap bi-encoder and trusting its top result blindly leaves exactly the precision problem this chapter opened with.