Skip to content
Lost in the Middle: How Language Models Use Long Contexts 4 min

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang · arXiv (Stanford, Berkeley, Samaya AI) 2023

Lost in the Middle: How Language Models Use Long Contexts

Hide the answer in a pile of documents and move it. Models read the start and the end well and lose the middle. See the U-shaped curve for yourself.

Read the original paper

AI & ML · White Belt · Pro · 4 min read

Lost in the Middle: How Language Models Use Long Contexts

Today's chatbots can take in enormous prompts: hundreds of pages in one go. It is tempting to treat that as a hard drive. Paste in everything, ask a question, and trust the model to find the right line.

A Stanford, Berkeley and Samaya AI team tested that trust in a simple way. They hid the answer to a question inside a pile of other documents, and moved it around. The model could read every document equally. Did it use them equally?

No. Models answered best when the useful passage was at the very start or the very end of the input, and worst when it sat in the middle, a U-shaped curve. With the answer in the middle, one popular model did worse than if it had been given no documents at all. Longer-context versions of models were not better at using the extra room.

How the test works

The main experiment is multi-document question answering. Each question comes from NaturalQuestions, a set of real search queries. The prompt holds a number of Wikipedia passages: 10, 20 or 30. Exactly one passage contains the answer. The rest are distractors that look related but do not. The authors then reorder the passages so the answer appears first, last, or anywhere in between, and measure how often the model answers correctly.

The paper's main test. For each of 2,655 questions, the documents were reshuffled so the passage with the answer landed in different positions, and the model's answer was scored each time.

They tested open models, MPT-30B-Instruct and LongChat-13B, and closed ones, OpenAI's GPT-3.5-Turbo and Anthropic's Claude-1.3, plus versions with longer context windows. Two reference points help. In the closed-book setting the model sees no documents. In the oracle setting it sees only the one document with the answer. For GPT-3.5-Turbo these scores were 56.1 percent and 88.3 percent.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

3 more sections, 1 more diagram and 2 interactive elements to play with

  • The U-shaped curve
  • More context is not always better
  • What to do about it

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.