Conversational RAG: Handling Follow-Ups
You'll learn to
- -Understand why a follow-up question breaks naive single-turn RAG retrieval
- -Implement query rewriting to resolve a follow-up into a standalone query
- -Recognize anaphora resolution as the underlying linguistic problem
A customer asks "What's my deductible?" and the bot answers correctly, retrieving their specific plan's information. The customer follows up: "And what about for dental?" The bot retrieves generic dental information, entirely losing the fact that the customer was asking about their specific plan's dental deductible, not dental deductibles in general. Every RAG pattern covered so far in this tier assumed a single, self-contained query. Real conversations are almost never that.
Why the Follow-Up Breaks Retrieval
"And what about for dental?" is a perfectly clear question to a human reading the full conversation, but read on its own, in isolation, exactly the way a retrieval system consumes it if nothing intervenes, it is nearly meaningless: what about dental, in relation to what? The referenced context ("my deductible," "my specific plan") lives entirely in the previous turn, not in the follow-up message's own text. Embedding "and what about for dental?" on its own and searching against it retrieves whatever is generically, superficially closest to those words, generic dental content, not the customer's specific plan.
Query Rewriting: Make the Follow-Up Stand on Its Own
The standard fix is query rewriting: before retrieval ever runs, use an LLM call (a small, fast, cheap one, this does not need the full-power model) to rewrite the follow-up into a standalone query that carries the necessary context from the conversation explicitly, in this case, "What is my dental deductible for my specific insurance plan?" That rewritten query is what actually gets embedded and searched, not the customer's original, context-dependent phrasing.
conversation_history = [
{"role": "user", "content": "What's my deductible?"},
{"role": "assistant", "content": "Your deductible is $1,500 for your Gold plan."},
]
follow_up = "And what about for dental?"
rewrite_prompt = f"""Given this conversation history:
{conversation_history}
Rewrite this follow-up question as a standalone question that makes sense
without needing the conversation history: "{follow_up}"
Standalone question:"""
# A well-rewritten result: "What is my dental deductible for my Gold plan?"
# THAT rewritten query, not the original follow-up, is what gets embedded
# and sent to the retriever.The Underlying Linguistics: Anaphora Resolution
The general linguistic name for this problem is anaphora resolution, figuring out what a pronoun or implicit reference ("it," "that," "what about X," "the other one") actually points back to. Query rewriting is, in effect, using an LLM to solve anaphora resolution as a side effect of restating the question clearly. This is also exactly why the ConversationMemory component from Tier 7 matters here: query rewriting needs the actual conversation history available to work from, it cannot resolve a reference to context it was never given.
Query rewriting adds a full extra model call before retrieval even starts, which adds both latency and cost to every single turn of a conversation. For a system with tight latency requirements, this is a real design tradeoff worth naming explicitly, not an invisible free improvement, and it is exactly the kind of place a small, fast, cheap model (recall the model-selection chapter from Tier 7) is the right tool, this task does not need frontier-model reasoning.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.