Entropy & KL Divergence
You'll learn to
- -Compute Shannon entropy and interpret it as the average surprise, or the number of yes/no questions needed
- -Compute KL divergence and explain why it is not symmetric, with numbers
- -Understand why RLHF penalizes a fine-tuned model with a KL-divergence term
- -Work out an RLHF-style penalized reward by hand
Two quantities from information theory come up constantly once you look past basic loss functions: entropy, which measures how uncertain a probability distribution is, and KL divergence, which measures how far one distribution has drifted from another. The second one specifically shows up again the moment you look at how a fine-tuned language model is kept from drifting too far from where it started.
An Analogy: Twenty Questions
In a game of twenty questions, you try to identify something with as few yes-or-no questions as possible. If there are four equally likely answers, you need two questions: one to halve them, and one to pick between the last two. If one answer is overwhelmingly likely, you can hardly ask anything: just check that guess. Entropy is the average number of such questions you need, which is a way of measuring how uncertain the situation is.
Entropy: How Spread Out Is a Distribution?
Entropy is maximized by a uniform distribution - maximum uncertainty, every outcome equally likely - and is zero for a distribution that puts all its probability on a single outcome, since there is nothing left to be uncertain about.
Run it and compare. The uniform distribution over 4 outcomes has entropy 1.3863, which is the natural log of 4. That number is in nats, because the code uses the natural logarithm. Using log base 2 instead gives bits: 1.3863 nats is exactly 2 bits, the two questions from the game. The near-certain distribution has entropy 0.1677 nats, about 0.24 bits: almost nothing left to ask. A fair coin has 1 bit.
KL Divergence: How Far Has q Drifted From p?
For KL divergence, imagine a codebook built for the wrong world. You expect events to follow q, so you give short codes to the ones q says are common. But the events really follow p. You spend extra effort on every message, and KL(p || q) is that extra effort per event, averaged over what actually happens. If q matches p, there is no waste and KL is zero.
KL divergence extends the same idea to compare two distributions instead of describing one. It is zero only when the two distributions are identical, and - this trips people up in interviews - it is not symmetric: KL(p || q) is generally not equal to KL(q || p).
Run it: KL(fine_tuned || reference) is 0.3681, and KL(reference || fine_tuned) is 0.5108. Same two distributions, different answers, because each direction asks a different question: how surprised are you when reality follows one, but you assumed the other? The two directions penalize different mistakes, which is why the order in the RLHF formula matters.
RLHF does not just maximize a reward model's score - left unconstrained, that reward-hacks into fluent-sounding nonsense. The fix subtracts beta * KL(policy || reference_model) from the reward being optimized: exactly the formula above, applied to a language model's own output distribution instead of two arbitrary ones. A larger beta keeps the fine-tuned model closer to its starting point; beta=0 removes the constraint entirely.
Working Out a Penalized Reward
Suppose a response earns a reward-model score of 5.0, the KL between the fine-tuned policy and the reference is 0.37 (close to the number above), and beta is 2. The score the optimizer actually sees is 5.0 - 2 x 0.37 = 4.26. If another response scores 5.4 but has drifted much further, say KL = 1.0, its score is 5.4 - 2.0 = 3.4. The penalty makes the drifted response worse, even though its raw reward is higher, which is how the KL term holds the model near its starting point.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Implement both formulas yourself: Level 45 (Entropy & KL Divergence) in AI Lab.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!