Skip to content
The Curse of Recursion: Training on Generated Data Makes Models Forget 4 min

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross Anderson · arXiv 2023, then Nature 2024

The Curse of Recursion: Training on Generated Data Makes Models Forget

What happens when AI learns from AI? Each copy forgets the rare cases. Run the bell-curve experiment from the paper yourself.

Read the original paper

AI & ML · White Belt · Pro · 4 min read

The Curse of Recursion: Training on Generated Data Makes Models Forget

Take a photocopy of a photograph. Now copy the copy. Do it again. By the tenth copy the faint details are gone, the dark areas have merged, and the picture looks like a smudge. Nothing was ever "broken". Each copy just lost a little.

A group of researchers from Oxford, Cambridge, Imperial College, Toronto and Edinburgh asked what happens to AI models when they are trained on text and images made by other AI models. The internet is filling with AI output, and the next models will learn from the internet. Their answer has a name: model collapse.

When each generation of a model learns from the previous generation's output, it slowly forgets the rare parts of the real world. First the unusual cases vanish, then the range of output narrows. The paper shows this in math, in simple models and in a language model, and concludes that access to real human-made data is essential.

A copy of a copy of a copy

The idea of model collapse in one picture. A model fits the data it saw and writes new data. The next model learns from that. Each round, the rare events become a little rarer, until the model covers only a narrow slice of what was real.

The paper defines model collapse as a degenerative process where the data a model generates ends up in the training set of the next model. Over generations the models forget the true underlying data. It differs from catastrophic forgetting, where one model forgets an old task while learning a new one. Here many models are linked in a chain, and each one misreads what is real.

Two kinds of error feed it. The main one is statistical error: a model sees only a finite number of examples, so something rare has a good chance of never being sampled. The second is functional error: the model is never a perfect copy of the data. The two errors add up over the chain.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

3 more sections, 1 more diagram and 2 interactive elements to play with

  • The simplest case: a bell curve
  • Does it happen in real models?
  • What to take from it

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.