The Model That Could Write But Wouldn't Listen
GPT-3 was impressive but frustrating to use. Ask it 'What is the capital of France?' and instead of answering 'Paris,' it might continue the pattern and generate more questions: 'What is the capital of Germany? What is the capital of Spain?' It was trained to predict text that looks like the internet, not to follow instructions. It could write poetry, but it could not reliably answer a direct question.
This gap between raw capability and actual usefulness was the central problem that RLHF (Reinforcement Learning from Human Feedback) solved. The model had the knowledge. It just did not know what you wanted it to do with that knowledge.
The Three-Step Process
InstructGPT (published by OpenAI in March 2022) used a three-step training process that turned a capable-but-uncooperative language model into one that actually followed instructions:
Step 1: Supervised Fine-Tuning (SFT)
Hire human contractors to write examples of ideal responses to prompts. 'What is the capital of France?' -> 'The capital of France is Paris.' Collect thousands of these demonstration pairs and fine-tune the pre-trained GPT-3 on them. This gives the model a basic understanding of the instruction-following format: user asks, model answers helpfully.
SFT alone gets you most of the way there. The model learns to generate helpful, on-topic responses. But it still produces some outputs that are verbose, evasive, or subtly wrong. And you cannot scale SFT indefinitely. Writing perfect demonstrations for every possible query is impossibly expensive.
Step 2: Train a Reward Model
Instead of writing perfect answers (expensive), have humans rank multiple model outputs from best to worst (cheaper). For a given prompt, generate 4-6 different responses from the SFT model, then have humans rank them. Use these rankings to train a separate model, the reward model, that predicts a scalar score for any prompt-response pair.
The reward model learns what humans consider 'good' responses: helpful, accurate, harmless, and appropriately detailed. It captures human preferences in a form that can be applied to millions of outputs automatically, without needing a human to evaluate each one.
Step 3: Optimize With PPO
Use the reward model as a scoring function and optimize the language model to generate responses that score highly. This is done using Proximal Policy Optimization (PPO), a reinforcement learning algorithm. The language model generates responses, the reward model scores them, and PPO adjusts the language model's weights to produce higher-scoring responses.
A crucial detail: the PPO optimization includes a KL divergence penalty that prevents the model from deviating too far from the original pre-trained model. Without this, the model could hack the reward model. Finding degenerate outputs that get high reward scores but are actually nonsensical. The KL penalty keeps the model grounded.
RLHF's core insight: it is easier for humans to judge quality than to produce it. Ranking 4 outputs takes 2 minutes. Writing the perfect output takes 20 minutes. RLHF converts cheap judgment into expensive behavior.
The Alignment Tax (and Why It's Worth Paying)
RLHF has costs. The model becomes slightly less capable at raw text completion. It loses some of the raw linguistic range of the base model. Researchers call this the 'alignment tax.' But the trade-off is overwhelmingly positive: a 1.3 billion parameter InstructGPT model was preferred by human raters over the 175 billion parameter base GPT-3. A model 100x smaller beat the larger one because it actually did what you asked.
This finding has a profound practical implication: capability alone is not enough. A model that follows instructions, stays on topic, admits uncertainty, and avoids harmful outputs is far more useful than a smarter model that ignores what you asked for. Every consumer AI product you use today. ChatGPT, Claude, Gemini. Uses some form of RLHF or its successors (DPO, RLAIF).
Beyond RLHF: DPO and RLAIF
RLHF is effective but complex. It requires training and maintaining a separate reward model, and PPO is notoriously unstable. Two important alternatives have emerged:
- DPO (Direct Preference Optimization): skips the reward model entirely. Instead of training a reward model from preferences and then doing RL, DPO directly optimizes the language model using preference pairs. Given a prompt with a preferred and dispreferred response, it adjusts weights to make the preferred response more likely. Simpler, more stable, increasingly popular.
- RLAIF (RL from AI Feedback): uses an AI model to generate the preference rankings instead of humans. Anthropic pioneered this with Constitutional AI, where Claude evaluates its own outputs against a set of principles. This dramatically reduces the cost of collecting feedback and scales to millions of evaluations.
What This Means for GenAI Engineers
RLHF matters for practical GenAI work in three ways.
First, every LLM you use in production has been through RLHF or a similar alignment process. When you compare GPT-4o vs GPT-4o-mini, part of the difference is in alignment quality. How well the model follows instructions, handles edge cases, and refuses harmful requests. Understanding RLHF helps you understand why models behave the way they do.
Second, prompt engineering works because of RLHF. The reason you can write 'You are a helpful customer support agent. Answer based on the provided context. If you don't know, say so.' and the model actually follows those instructions is because RLHF trained it to be instruction-following. Without RLHF, those instructions would be ignored.
Third, the FeedbackCollector component in GenAI pipelines is a lightweight version of the same loop. When users give thumbs up/down on responses, you are collecting the same kind of preference data that RLHF uses. Over time, this data can be used to fine-tune or evaluate your pipeline's output quality.
Try It in the Lab
In the GenAI Pipeline Lab, notice how the PromptTemplate component matters more than you might expect. RLHF made models instruction-following, but the quality of your instructions still determines the quality of the output. Write clear, specific system prompts and see how the evaluation scores change.
Further Reading
- Ouyang et al. (2022): 'Training language models to follow instructions with human feedback,' the InstructGPT/RLHF paper
- Christiano et al. (2017): 'Deep reinforcement learning from human preferences,' the original RLHF paper
- Rafailov et al. (2023): 'Direct Preference Optimization,' the DPO paper that simplifies RLHF
- Bai et al. (2022): 'Constitutional AI,' Anthropic's approach to AI alignment using AI feedback
- Lilian Weng's 'Reward Hacking in Reinforcement Learning,' understanding how RL can go wrong
