Skip to content
GenAI Learn/Vector Search Fundamentals
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Reranking for Precision

4 min read

You'll learn to

  • -Explain why a first retrieval pass puts the best document at position five to eight too often
  • -Contrast bi-encoders (fast, precomputed) with cross-encoders (slow, accurate, run per pair)
  • -Show with numbers why reranking is a second stage on a shortlist and not a replacement for vector search

A medical RAG system retrieves ten chunks for every query, but the truly relevant one often sits at position five through eight, not one. For a doctor scanning results under pressure, or for an LLM given all ten chunks with no hint of which matters most, that buried ranking is a real and sometimes urgent problem.

An Analogy: The CV Screen and the Interview

A company with ten thousand job applicants cannot interview them all. A quick screen of each CV picks fifty promising names. That screen is fast but crude, because it never sees the applicant and the role together. The company then interviews the fifty properly, reading each person against the job. The interview is slow but much more accurate. Retrieval and reranking work the same way.

Bi-Encoders: Fast, and a Little Imprecise

The embedding models used for the first retrieval pass, covered in the last two chapters, are bi-encoders. The query and every document are embedded completely independently, with no interaction, and then compared by a simple, fast operation such as cosine similarity. That independence is what makes vector search fast enough for millions of documents: every document's embedding is computed once, ahead of time, and a new query is embedded once and compared. The price is precision. The model never reads the query and a specific document together, so it can miss relevance that only shows when the two are considered jointly.

Cross-Encoders: Slow, and Much More Precise

A cross-encoder takes the query and one candidate document together, as a single combined input, and outputs a direct relevance score for that pair. Reading both texts jointly, in one forward pass, captures interactions a bi-encoder's independent embeddings structurally cannot. But it must be run once per candidate, with nothing precomputed, which makes it far too slow to run over a corpus of millions of documents. Try the interactive version below to see the two orderings side by side.

Query: "Is it safe to take aspirin with my Warfarin prescription?"

10 chunks retrieved by fast vector search. The actual answer isn't at the top.

1General overview of blood thinner medications
2Common side effects of anticoagulant therapy
3History of anticoagulant drug development
4Dosage guidelines for elderly patients
5Pharmacy inventory management for anticoagulants
6Warfarin and aspirin: dangerous interaction and bleeding risk← the actual answer
7Patient monitoring schedule for blood thinners
8Insurance coverage for anticoagulant prescriptions
9Comparing generic vs. brand-name blood thinners
10Anticoagulant clinic scheduling procedures

Vector search alone found the right neighborhood, but ranked by rough similarity, not by carefully reading each chunk against the exact question.

Why You Need Both, in Two Stages

This is why reranking is a second stage on top of retrieval, not a replacement. Vector search, cheap and precomputed, narrows millions of documents to a shortlist, typically the top 20 to 100. A cross-encoder reranks only that shortlist, applying its expensive but accurate scoring where it can afford to. The demo puts numbers on it, using an illustrative 0.02 seconds per cross-encoder pass.

Press Run: reranking everything takes hours, reranking a shortlist takes about a second
Precision vs. Recall Across the Two Stages
Stage 1 (bi-encoder)
Optimizes for recall: do not miss the right document
Stage 2 (cross-encoder)
Optimizes for precision: get the ordering of the shortlist right
~10-100x slower
Roughly, cross-encoder scoring cost vs. bi-encoder, per document
20-100 candidates
A typical reranking shortlist size, small enough to afford the cost

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Build the Vector Vault, Hybrid Searcher, and Reranker levels in the GenAI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!