Skip to content
Language Models are Few-Shot Learners 3 min

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam · NeurIPS 2020

Language Models are Few-Shot Learners

The GPT-3 paper explained: in-context learning, zero, one and few-shot prompting, and why scale matters. Build a prompt and price a 175-billion-parameter model.

Read the original paper

AI & ML · Yellow Belt · Pro · 3 min read

Language Models are Few-Shot Learners

Until 2020, the standard way to get a language model to do a job was to fine-tune it: take a pre-trained model and keep training it on thousands of labeled examples for that one task. It works, but it is awkward. Every new task needs its own labeled dataset. And humans do not need that. Show a person two or three examples of a new task, or just describe it, and they can usually do it.

OpenAI's GPT-3 paper explored whether a very large language model could do the same. The model has 175 billion parameters, ten times more than any earlier non-sparse language model, according to its authors. Its striking finding is that it can pick up many tasks from the examples you put in the prompt, with no further training at all.

When a language model is big enough, you can teach it a task inside the prompt. Describe the task, add a few examples, and let it continue the pattern. The paper calls this in-context learning, and it works better as models get larger.

Zero-shot, one-shot and few-shot

GPT-3 is an autoregressive language model: it reads text and predicts the next word, over and over. The paper tests it in three settings and contrasts them with fine-tuning. In all three, no weights are updated.

  • Zero-shot: only a plain-language instruction. Most convenient, but also the hardest for the model.
  • One-shot: an instruction and one worked example.
  • Few-shot: an instruction and several worked examples. The paper fits as many as the context window allows, typically 10 to 100.
Fine-tuning changes the model's weights using many labeled examples. GPT-3's three settings change nothing inside the model. The examples live in the prompt and the model reads them at run time.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 3 interactive elements to play with

  • Size is the ingredient
  • What the authors warn about

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.