PPO, RLHF & How Reasoning Models Are Trained
You'll learn to
- -Explain, with the probability ratio and the advantage, why an unconstrained policy update can wreck a large model
- -Compute PPO's clipped objective for three cases and say which one is not protected by the clip
- -Describe the three steps of RLHF and what reward hacking is
- -Compute a KL-penalized reward and a group-relative advantage as used by GRPO
The previous chapter ended with a policy, the model's way of choosing what to do (for an LLM, how likely it is to produce each next word), and with the advantage, which says how much better than expected an answer was. This chapter covers the two safety ideas that make training on those signals practical for large language models: PPO's clipped update, and a KL penalty that stops a fine-tuned model drifting into nonsense. Combined in a recipe called RLHF, reinforcement learning from human feedback, they are how chat models become helpful.
An Analogy: A Cautious Driving Instructor
After each lesson the instructor lets you adjust your driving habits, but only a little. One lucky lap does not justify rewriting your whole style, and a bold change that works once may be a disaster in the rain. A second rule keeps you recognizably yourself: however much you improve, you must not drive in a way that no longer resembles how you started.
- -Your driving style is the policy.
- -One lesson is a batch of training examples.
- -How far your new style differs from the old one is the probability ratio.
- -The instructor's limit on change per lesson is PPO's clip.
- -The rule that you must still resemble your starting self is the KL penalty.
The Ratio and the Advantage
Pick one action the model took. The ratio r is the probability the new policy gives that action divided by the probability the old policy gave it. If the old policy chose it with probability 0.20 and the new one with 0.30, then r = 0.30 / 0.20 = 1.5. A ratio of 1 means nothing changed. The advantage A, from the previous chapter, is positive when the action beat expectations and negative when it did worse.
The obvious training objective is r x A: raise the probability of good actions and lower it for bad ones. The danger is that a single unusual batch can push r far from 1, making one enormous step. For a small robot policy that is a bad episode. For a large language model a wrecked update is expensive to undo, which is exactly why Proximal Policy Optimization (PPO) exists.
PPO's Clipped Objective
Run the three cases below and compare.
- -Ratio 1.0, advantage +2: the policy has not moved, so the objective is simply 2.0.
- -Ratio 1.5, advantage +2: the unclipped value would be 3.0, but the ratio is capped at 1.2, so the objective is 2.4. Pushing the probability up by more than 20 percent earns no extra credit, so there is no reason to jump.
- -Ratio 1.5, advantage -2: the unclipped value is -3.0 and the minimum keeps it. A bad action that became more likely is not protected by the clip, and training keeps the full penalty so it can be corrected.
That last case is the subtle one. The clip only removes the incentive to push a good update (positive advantage) too far. If an action's advantage is negative, clipping does not protect the objective from getting even more negative. A genuinely bad update is never let off the hook.
The demo below plots the objective against the ratio. Try a good action with a large ratio, then a bad action with the same ratio, and compare where the clip protects the update and where it does not.
PPO's Clip: Where It Protects and Where It Doesn't
The ratio says how much likelier the new model makes an action. The advantage says how good that action was. Move both and watch the clipped objective, drawn as the solid line, against the unclipped one, drawn dashed.
Unclipped (r × A)
3.00
PPO objective
2.40
Clip changed it?
yes
Good action made much likelier (ratio 1.50, above the cap 1.20): the objective is held at 2.40 instead of 3.00. Pushing the probability up any further earns nothing, so there is no reason to jump.
RLHF in Three Steps
- -Supervised fine-tuning. Show the pretrained model many good example conversations so it learns the format of being an assistant.
- -Train a reward model. Humans rank pairs of answers to the same prompt, and a model learns to predict which answer they would prefer.
- -Optimize the policy with PPO against that reward model, keeping a copy of the starting model as a frozen reference.
Step 3 has a trap. The reward model is only a stand-in for what people actually want, so a policy that chases its score hard enough will find tricks the stand-in likes but people do not: answers that are long, confident, flattering and wrong. This is reward hacking. The fix is a leash.
The KL Penalty: A Leash on the Policy
KL divergence is a number that measures how different two probability distributions are. It is 0 when they are identical and grows as they drift apart. RLHF subtracts it from the reward, so the further the policy strays from the frozen reference model, the more reward it loses.
A concrete case. The reference model is undecided between two words, [0.5, 0.5]. After some training the policy prefers the first word, [0.9, 0.1]. Their KL divergence is about 0.368. With a reward-model score of 1.0 and beta = 0.5, the reward that training sees is 1.0 - 0.5 x 0.368 = 0.816. With beta = 0 the leash is off and it sees the full 1.0. Run it to confirm.
GRPO: Judging Each Answer Against Its Own Group
Training a value function to estimate the baseline is costly. GRPO, Group Relative Policy Optimization, used to train DeepSeek-R1 and many reasoning models since, skips it. For each prompt it samples a group of answers, scores them all, and judges each answer against the group's own average. With scores [1, 0, 0, 1] (1 means correct), the mean is 0.5 and the standard deviation is 0.5, so the advantages are +1, -1, -1 and +1. The correct answers are reinforced, the wrong ones suppressed.
Everything else carries over from this chapter. GRPO and its relatives still use the clipped ratio to keep each update small and the KL penalty to keep the model close to its reference. Only the source of the advantage changes: a group of sampled answers instead of a separately trained value function.
AI Lab's Level 59 (PPO's Clipped Surrogate Objective) and Level 60 (KL-Penalized Reward) implement both formulas directly in numpy. Level 60 reuses the exact KL divergence function you built all the way back in Level 45, now applied to a policy versus a reference model instead of two arbitrary distributions.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Implement both from scratch: Level 59 (PPO's Clipped Surrogate Objective) and Level 60 (KL-Penalized Reward) in AI Lab.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!