LLM Hallucination and RAG: Why AI Makes Things Up and How Retrieval Fixes It

Your AI Just Made Something Up. Confidently.

Here is a real scenario. A developer builds a chatbot for a law firm. A client asks about a specific contract clause. The LLM generates a beautifully written response citing Section 7.3 of the Master Services Agreement. There is just one problem: Section 7.3 does not exist. The model invented it. The response was grammatically perfect, used the right legal terminology, and felt completely authoritative. None of it was real.

This is hallucination. It is the tendency of language models to generate text that is fluent, confident, and completely wrong. And it is not a bug that can be patched. It is a fundamental property of how these models work. Understanding why it happens is the first step toward building systems that handle it.

Why LLMs Hallucinate

Language models are trained to predict the most probable next token. They are not trained to be truthful. When you ask GPT-4 about your company's refund policy, it does not search for the answer. It generates the most statistically likely continuation of the text. If many training documents describe refund policies in a certain way, the model produces something that looks like a refund policy. Whether it matches YOUR refund policy is irrelevant to the model. It was never asked to be accurate. It was asked to be probable.

There are three distinct types of hallucination, and each one creates different problems in production:

• Factual hallucination: inventing facts, statistics, or citations that never existed. The model generates a paper title that sounds plausible but no such paper was ever written. It cites a court case that never happened. It produces statistics that are entirely fabricated.
• Faithfulness hallucination: contradicting the source material provided in context. You give the model a document saying the return window is 30 days, and it tells the user the return window is 60 days. The information was right there in the prompt, but the model overrode it with its training distribution.
• Intrinsic hallucination: generating self-contradictory text within the same response. The model says 'the meeting is on Tuesday' in paragraph one and 'the Thursday meeting' in paragraph three. It contradicts itself because each token prediction is locally optimal without global consistency checking.

Hallucination is not the model being confused. It is the model doing exactly what it was trained to do: generating probable text. Probable and true are not the same thing.

Why This Problem Is So Dangerous

For a consumer chatbot, hallucination is annoying. Someone asks for restaurant recommendations and gets a place that closed two years ago. Mildly frustrating, nobody gets hurt.

For enterprise applications, hallucination is a dealbreaker. A legal AI that fabricates case citations exposes the firm to malpractice. A medical AI that invents drug interactions could harm patients. A financial AI that hallucinates compliance requirements could lead to regulatory violations. In these domains, being confidently wrong is far worse than admitting uncertainty.

The insidious part is that hallucinated text looks exactly like real text. There is no signal in the output that distinguishes truth from fabrication. The model uses the same confident tone whether it is stating a verified fact or inventing one from scratch. You cannot just read the output and tell which parts are real. That is what makes this problem so hard and why the industry needed a systematic solution.

RAG: Giving the Model an Open Book Exam

In May 2020, Lewis et al. at Meta AI published the paper that launched a thousand startups. Their idea was elegant: instead of asking the model to answer from memory, give it the relevant documents and ask it to answer from those. Think of it like the difference between a closed-book exam and an open-book exam. In a closed-book exam, you rely on what you memorized, and you might misremember details. In an open-book exam, you can look up the answer, making you far more accurate.

The Retrieval-Augmented Generation (RAG) pipeline works in five steps:

1. User asks a question
2. The system searches a knowledge base for the most relevant passages using semantic similarity
3. Those retrieved passages are inserted into the prompt as context, giving the model the actual source material
4. The LLM generates an answer grounded in the provided context, not just its training data
5. The answer can cite specific passages, making it verifiable by humans

This simple pattern reduces hallucination dramatically. The model is no longer guessing from its training distribution. It is synthesizing information from documents you control. You can update the knowledge base in real time without retraining the model. You can verify the answer against the cited sources. And when the model does not find relevant information, you can have it say 'I don't know' instead of making something up.

The RAG Architecture in Detail

A production RAG system has two distinct phases. Understanding both is critical because the offline phase determines what your system CAN find, and the online phase determines what it DOES find.

The Indexing Phase (Offline)

Before your system can answer questions, you need to prepare your knowledge base. This happens offline, usually as a batch job:

1. Collect documents from your data sources (PDFs, wikis, databases, APIs)
2. Split documents into chunks. Typical sizes range from 256 to 512 tokens. Too large and the chunks contain irrelevant information. Too small and they lack context.
3. Embed each chunk into a vector using an embedding model like OpenAI's text-embedding-3 or open-source alternatives like BGE or E5
4. Store the vectors in a vector database (Pinecone, Weaviate, Qdrant, pgvector) with the original text as metadata

The Query Phase (Online)

When a user asks a question, the system executes this pipeline in real time:

1. Embed the user's question using the same embedding model
2. Search the vector database for the K most similar chunks (typically K = 3 to 10)
3. Construct a prompt that includes the retrieved chunks as context along with the user's question
4. Send the prompt to the LLM, which generates an answer based on the provided context
5. Optionally post-process the response to add citations, check for hallucination, or filter unsafe content

Every component in this pipeline has configuration choices that compound. The chunking strategy determines passage granularity. The embedding model determines search quality. The value of K balances context richness against noise. The prompt template determines how well the model uses the context. Each decision matters, and getting them right is what separates a demo RAG system from a production one.

What RAG Does Not Solve

RAG is not a silver bullet. It significantly reduces hallucination, but it does not eliminate it. The model can still hallucinate in several ways even with RAG:

• If the retriever fails to find the right passages, the model may answer from its training data instead of admitting ignorance
• The model might synthesize information from multiple chunks in ways that create new inaccuracies, combining true facts into a false conclusion
• Long retrieved passages can overwhelm the model's attention, causing it to focus on irrelevant parts
• The model might add qualifiers or details from its training data that are not present in the retrieved context

This is why production RAG systems need additional layers: guardrails to detect hallucination, evaluation frameworks to measure accuracy, and human feedback loops to identify failure modes. Basic RAG gets you 80% of the way there. The remaining 20% requires the advanced techniques covered in the next two blog posts.

Try It in the Lab

Start with Act 2, Level 11. Build a basic RAG pipeline: FileLoader, Chunker, EmbeddingModel, VectorDB, PromptTemplate, LLM, OutputParser. This pattern solves 80% of real-world GenAI use cases. Pay attention to how each component's configuration affects the architecture score. The lab teaches you the same trade-offs you will face in production.

Further Reading

• Lewis et al. (2020): 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks'
• Ji et al. (2023): 'Survey of Hallucination in Natural Language Generation'
• Huang et al. (2023): 'A Survey on Hallucination in Large Language Models'
• LangChain RAG tutorial for practical implementation