Skip to content
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks 4 min

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela · NeurIPS 2020

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

The paper that named RAG: a retriever plus a generator, with knowledge you can update by swapping an index. Try a miniature retrieve-then-answer pipeline.

Read the original paper

AI & ML · Yellow Belt · Pro · 4 min read

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Ask a large language model a question about something it learned during training, and it often answers well. Ask about last week's news, your company's policies, or an obscure fact, and it may answer confidently and wrongly. The knowledge a model has is frozen into its weights. It is hard to check where an answer came from, and to update a fact you would have to retrain.

This 2020 paper from Facebook AI Research, University College London and NYU proposed a simple remedy that has become the standard pattern for AI products that must stay accurate: retrieval-augmented generation, or RAG. Before answering, look up relevant documents, and let the model write its answer with those documents in front of it.

RAG combines a language model's built-in knowledge (parametric memory) with a searchable document store (non-parametric memory). For each question it retrieves the most relevant passages, then generates an answer conditioned on them. The knowledge can be updated by changing the index, not the model.

Two kinds of memory

The paper opens with a limitation. Pre-trained language models store factual knowledge in their parameters, but their ability to access and precisely use that knowledge is limited. They also cannot easily show where an answer came from or be updated when the world changes. Earlier systems that did look things up were mostly built for narrow tasks, such as extracting an answer span from a passage.

The paper's framing. A language model keeps knowledge in its parameters (parametric memory). RAG adds a searchable store of documents (non-parametric memory) that it can consult for every question.

The authors build a general-purpose version. It works for any task where the output is text, from answering open questions to writing a short fact-checked statement.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • How a RAG model works
  • What the paper found

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.