Skip to content
Why Language Models Hallucinate 4 min

Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang · arXiv (OpenAI, Georgia Tech) 2025

Why Language Models Hallucinate

OpenAI researchers argue chatbots bluff because tests reward guessing. Try the scoring simulator and see why an honest model can lose a leaderboard.

Read the original paper

AI & ML · White Belt · Pro · 4 min read

Why Language Models Hallucinate

Ask a chatbot for the birthday of someone obscure and it may give you a date, with total confidence, even though it cannot know. The authors of this paper tried exactly that. A leading open model answered three times with three different wrong dates, even though the prompt said to answer only if it knew.

People call this hallucination. It can feel like a mysterious flaw. The paper argues it is not mysterious at all. It has two plain causes, and one of them is how we grade the models.

Hallucinations come from two things. First, facts a model saw only once are statistically hard to tell from plausible falsehoods. Second, most tests give a point for a right answer and nothing for "I don't know", so guessing always scores at least as well as admitting doubt. Models trained and ranked on those tests learn to bluff.

Cause one: facts the model barely saw

During pretraining a model reads huge amounts of text. Some facts show up thousands of times and are learned well. Others, such as one person's birthday, appear in the data exactly once, or never.

The paper shows that a model cannot reliably separate such a rare fact from a convincing wrong one. Mathematically it turns text generation into a yes or no question: "Is this a valid answer?" The authors prove that the rate of generated errors is at least twice the rate at which a classifier gets that question wrong.

The paper's own example. A fact seen only once looks no different from a plausible guess, so the model cannot tell it apart. If one fact in five is seen once, expect at least one wrong answer in five.

There is a simple rule of thumb. If 20 percent of the birthday facts appeared exactly once in training, expect the model to hallucinate on at least 20 percent of such birthdays. The paper calls the share of once-seen facts the singleton rate.

Imagine memorizing a phone book by reading each page one time. You will know the famous numbers you saw many times. For a name you saw once, you can produce a believable number, but you cannot check it. Without a way to check, a believable number and a true number feel the same.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • Cause two: tests that reward bluffing
  • The fix: change the scoreboards

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.