Ask the original GPT-3 for a recipe and it might continue your sentence with more recipe-sounding text. Ask it to "explain the moon landing to a six-year-old" and it might produce a list of similar-sounding questions. It was trained to predict the next word on the Internet, not to do what you asked. The OpenAI authors write that making language models bigger does not inherently make them better at following a user's intent.
InstructGPT is the paper that showed how to fix this by training on human feedback. Its method, now called RLHF (reinforcement learning from human feedback), is behind the leap from raw language models to the assistants people actually use.
Take a pre-trained model, show it good answers, train a reward model from human rankings of answers, then use reinforcement learning to make the model produce answers the reward model scores highly. The resulting model follows instructions far better, even at a much smaller size.
The alignment gap
A language model trained only to predict text has no built-in idea of what a user wants. The paper lists what goes wrong: outputs that are untruthful, toxic, or simply not helpful. The authors call a model aligned if it acts in accordance with the user's intention, including the unstated intentions such as staying truthful.
Better prompting helps (see the few-shot paper), but it is a patch. The authors want the model itself to prefer helpful behavior. Since "helpful" cannot be written as a formula, they learn it from people.
Three steps
- Supervised fine-tuning (SFT). Human labelers write ideal responses to a collection of prompts. GPT-3 is fine-tuned to imitate them. This teaches the model the basic habit of following instructions.
- Reward model (RM). For each prompt, the model produces several answers, and labelers rank them. A separate model is trained to predict which answer people prefer, and outputs a single score. In the paper, labelers ranked between 4 and 9 answers at a time, which gives many pairwise comparisons from each ranking.
- Reinforcement learning (PPO). The model writes answers, the reward model scores them, and the model is updated to earn higher scores, using an algorithm called PPO. Steps 2 and 3 can be repeated with newer answers.
