Until 2020, the standard way to get a language model to do a job was to fine-tune it: take a pre-trained model and keep training it on thousands of labeled examples for that one task. It works, but it is awkward. Every new task needs its own labeled dataset. And humans do not need that. Show a person two or three examples of a new task, or just describe it, and they can usually do it.
OpenAI's GPT-3 paper explored whether a very large language model could do the same. The model has 175 billion parameters, ten times more than any earlier non-sparse language model, according to its authors. Its striking finding is that it can pick up many tasks from the examples you put in the prompt, with no further training at all.
When a language model is big enough, you can teach it a task inside the prompt. Describe the task, add a few examples, and let it continue the pattern. The paper calls this in-context learning, and it works better as models get larger.
Zero-shot, one-shot and few-shot
GPT-3 is an autoregressive language model: it reads text and predicts the next word, over and over. The paper tests it in three settings and contrasts them with fine-tuning. In all three, no weights are updated.
- Zero-shot: only a plain-language instruction. Most convenient, but also the hardest for the model.
- One-shot: an instruction and one worked example.
- Few-shot: an instruction and several worked examples. The paper fits as many as the context window allows, typically 10 to 100.
