Skip to content
LLMs Get Lost In Multi-Turn Conversation 4 min

Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville · arXiv 2025, then ICLR Outstanding Paper 2026

LLMs Get Lost In Multi-Turn Conversation

An award-winning ICLR 2026 study: the same information, spoken in pieces, makes chatbots about a quarter worse. See why, and what to do about it.

Read the original paper

AI & ML · White Belt · Pro · 4 min read

LLMs Get Lost In Multi-Turn Conversation

Most tests of language models give the model one complete question and score the answer. Real conversations are not like that. People start with half an idea, add details as they think of them, and correct themselves along the way. A chat is a conversation, and it is often a messy one.

This paper asks a simple question: does a model do as well when it receives the same information a little at a time? The answer, across fifteen well-known models, is no. The paper won an Outstanding Paper award at ICLR 2026.

The researchers took tasks that models do well in one message and fed them piece by piece, one piece per turn. Average performance fell by about a quarter. The models were not much less capable. They were far less reliable, because they answered too early and then stuck to their first guess.

How the test works

The team started from existing benchmarks across six kinds of work: writing code, database queries, API actions, math, describing data in text, and summaries. They turned each instruction into a list of small pieces that they call shards. Each shard holds one fact from the original instruction.

The paper's test. A normal question is cut into small pieces, and one piece is revealed per turn. Nothing is hidden: by the last turn the model has heard everything it would have heard at once. The pieces here are shortened from the paper's example.

A simulated user, itself an AI, then revealed one shard per turn and replied naturally to the model. Nothing was hidden or changed in meaning. In all, the study ran more than 200,000 simulated conversations. It also ran the single-message version of each task as a baseline, so any difference comes from how the information arrived.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

3 more sections, 1 more diagram and 2 interactive elements to play with

  • The result: a quarter of the score disappears
  • Why models get lost
  • What it means for builders and for you

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.