Advanced RAG: Hybrid Search, Reranking, and How to Actually Evaluate Quality

Basic RAG Is Not Enough

You built a RAG pipeline. It works great on your demo dataset. Then real users arrive and everything breaks. Someone asks using domain jargon that the embedding model has never seen. Another asks a multi-hop question that requires connecting information from three different documents. A third asks a question where the answer is in a table, but your chunker split the table across two chunks and neither half makes sense.

Basic RAG (embed, retrieve, generate) works for about 60-70% of real-world queries. The remaining 30-40% require more sophisticated approaches. That gap is the difference between a demo and a product. Closing it requires three things: better search, smarter retrieval patterns, and systematic evaluation.

Hybrid Search: Combining Two Kinds of Smart

Pure vector search is great at understanding meaning. Someone searching for 'return policy' will find passages about 'refund procedures' because the embeddings capture semantic similarity. But vector search struggles with precise terms. Search for 'error code 7842' and the embedding model might return passages about error handling in general, missing the one passage that mentions that specific code.

Pure keyword search (BM25) is the opposite. It excels at exact matches. Search for 'error code 7842' and it will find every passage containing that exact string. But it fails at semantic understanding. 'How to fix login issues' will not match a passage about 'authentication failure troubleshooting' because the words are different.

Hybrid search combines both approaches, and the results are consistently better than either one alone. There are several ways to combine them:

• Reciprocal Rank Fusion (RRF): run both searches independently, then merge the ranked results. A passage that ranks highly in both lists gets boosted. Simple to implement and remarkably effective.
• Weighted combination: assign weights like 0.7 for semantic and 0.3 for keyword, then combine the scores. Requires tuning the weights for your specific data, but gives you more control.
• Cascaded search: use keyword search first to narrow the candidate set, then apply semantic search on the candidates. Efficient for very large document collections.

In practice, hybrid search improves retrieval accuracy by 5-15% over pure semantic search. The exact improvement depends on your data. If your documents contain lots of specific identifiers (product codes, legal references, technical terms), the keyword component helps a lot. If your queries are mostly conversational, the semantic component does most of the work.

Reranking: The Second Opinion That Changes Everything

Embedding models are bi-encoders. They encode the query and each passage independently, then compare the vectors. This is fast (you can compare against millions of pre-computed vectors) but imprecise, because the query and passage never directly interact during encoding.

Cross-encoder rerankers take a different approach. They process the query and passage TOGETHER as a single input, allowing deep token-level interaction between them. This is dramatically more accurate but also much slower, because you cannot pre-compute passage encodings.

The production pattern is a two-stage pipeline:

1. First stage (recall): use fast embedding search to retrieve 20-50 candidate passages from the full collection
2. Second stage (precision): use a cross-encoder reranker to precisely score each candidate against the query, then keep the top 3-5

You get the speed of embedding search (scanning millions of passages) with the precision of cross-attention (deeply comparing each candidate). Adding a reranker typically improves answer quality by 10-20%, which is often a bigger improvement than switching to a more expensive LLM.

Popular rerankers include Cohere Rerank, BGE Reranker, and cross-encoder models from HuggingFace. Some vector databases like Weaviate have built-in reranking support.

Advanced RAG Patterns

Beyond hybrid search and reranking, the research community has developed increasingly sophisticated retrieval strategies. Each one solves a specific failure mode:

Query Transformation

Sometimes the user's query is not the best search query. 'Why is my app crashing?' is a terrible search query because it is too vague. Query transformation uses an LLM to rewrite the query before searching. One powerful variant is HyDE (Hypothetical Document Embeddings): the LLM generates a hypothetical answer to the question, and then you search for real passages similar to that hypothetical answer. Since the hypothetical answer is in the same 'language' as the documents, it often retrieves better results than the original question.

Multi-hop Retrieval

Some questions require connecting information from multiple passages. 'Which team lead manages the project with the highest budget?' requires first finding the highest budget project, then finding who manages it. Multi-hop retrieval chains multiple search steps, using the results from each step to formulate the next query. Each hop narrows down the information until the final answer emerges.

Parent Document Retrieval

Index small chunks (128 tokens) for precise search but return the larger parent chunk (512-1024 tokens) to the LLM. This gives you the best of both worlds: precise matching finds the right location, but the LLM gets enough surrounding context to generate a complete answer. Particularly useful for technical documentation where a specific detail needs its surrounding context to make sense.

Corrective RAG and Self-RAG

What if the retriever fails? Corrective RAG adds a quality check after retrieval. If the retrieved passages score below a confidence threshold, the system falls back to web search or asks the user to clarify. Self-RAG goes further: it lets the model decide WHEN to retrieve (not every question needs retrieval) and WHAT to retrieve (the model can generate its own search queries). This makes the system more adaptive but also more complex to build and evaluate.

Evaluating RAG Quality

You cannot improve what you cannot measure. RAG evaluation is still an evolving field, but frameworks like RAGAS have established useful metrics:

• Context Precision: of the passages retrieved, what fraction are actually relevant? High precision means less noise in the context.
• Context Recall: of all the relevant passages in your knowledge base, what fraction did the retriever find? High recall means important information is not being missed.
• Faithfulness: does the generated answer stick to the facts in the retrieved context, or does the model add information from its training data? Low faithfulness indicates hallucination.
• Answer Relevancy: does the answer actually address the user's question? A response can be factually correct but fail to answer what was asked.
• Answer Correctness: compared to a ground truth answer, how accurate is the generated response?

To evaluate your RAG system, create 50-100 test queries with ground truth answers. Run your pipeline on each query, compute the metrics, and look for patterns. Most RAG failures are retrieval failures. If the right passages reach the LLM, it almost always generates a good answer. If the retriever misses the right passages, no amount of prompt engineering will fix the response.

Most RAG failures are retrieval failures, not generation failures. If the right passages reach the LLM, it almost always generates a good answer. Fix retrieval first, everything else second.

Putting It All Together

A production-grade RAG pipeline typically looks like this: the user's query goes through query transformation, then hybrid search (semantic + BM25) retrieves 30 candidates, a reranker scores and filters to the top 5, those passages are assembled into a prompt with the original question, and the LLM generates a cited answer. Guardrails check the output for hallucination and safety, and the entire trace is logged for evaluation.

You do not need all of these components on day one. Start with basic RAG. Measure where it fails. Add hybrid search if keyword queries are underperforming. Add a reranker if the right passages are being retrieved but ranked too low. Add query transformation if users' natural language does not match your document style. Each component addresses a specific, measurable failure mode.

Try It in the Lab

Act 2 introduces Rerankers and advanced retrieval components. Notice how adding a Reranker improves the architecture score but adds latency. The lab's scoring system reflects the real production trade-off: better answers take more time and cost more compute. Learning to balance these trade-offs is the core skill of a GenAI engineer.

Further Reading

• Gao et al. (2023): 'Retrieval-Augmented Generation for Large Language Models: A Survey'
• Es et al. (2023): 'RAGAS: Automated Evaluation of Retrieval Augmented Generation'
• Gao et al. (2022): 'Precise Zero-Shot Dense Retrieval without Relevance Labels' (HyDE paper)
• LlamaIndex advanced RAG guide
• Anthropic RAG best practices documentation