Reranking for Precision
You'll learn to
- -Explain why a first retrieval pass puts the best document at position five to eight too often
- -Contrast bi-encoders (fast, precomputed) with cross-encoders (slow, accurate, run per pair)
- -Show with numbers why reranking is a second stage on a shortlist and not a replacement for vector search
A medical RAG system retrieves ten chunks for every query, but the truly relevant one often sits at position five through eight, not one. For a doctor scanning results under pressure, or for an LLM given all ten chunks with no hint of which matters most, that buried ranking is a real and sometimes urgent problem.
An Analogy: The CV Screen and the Interview
A company with ten thousand job applicants cannot interview them all. A quick screen of each CV picks fifty promising names. That screen is fast but crude, because it never sees the applicant and the role together. The company then interviews the fifty properly, reading each person against the job. The interview is slow but much more accurate. Retrieval and reranking work the same way.
Bi-Encoders: Fast, and a Little Imprecise
The embedding models used for the first retrieval pass, covered in the last two chapters, are bi-encoders. The query and every document are embedded completely independently, with no interaction, and then compared by a simple, fast operation such as cosine similarity. That independence is what makes vector search fast enough for millions of documents: every document's embedding is computed once, ahead of time, and a new query is embedded once and compared. The price is precision. The model never reads the query and a specific document together, so it can miss relevance that only shows when the two are considered jointly.
Cross-Encoders: Slow, and Much More Precise
A cross-encoder takes the query and one candidate document together, as a single combined input, and outputs a direct relevance score for that pair. Reading both texts jointly, in one forward pass, captures interactions a bi-encoder's independent embeddings structurally cannot. But it must be run once per candidate, with nothing precomputed, which makes it far too slow to run over a corpus of millions of documents. Try the interactive version below to see the two orderings side by side.
Query: "Is it safe to take aspirin with my Warfarin prescription?"
10 chunks retrieved by fast vector search. The actual answer isn't at the top.
Vector search alone found the right neighborhood, but ranked by rough similarity, not by carefully reading each chunk against the exact question.
Why You Need Both, in Two Stages
This is why reranking is a second stage on top of retrieval, not a replacement. Vector search, cheap and precomputed, narrows millions of documents to a shortlist, typically the top 20 to 100. A cross-encoder reranks only that shortlist, applying its expensive but accurate scoring where it can afford to. The demo puts numbers on it, using an illustrative 0.02 seconds per cross-encoder pass.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Build the Vector Vault, Hybrid Searcher, and Reranker levels in the GenAI Lab.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!