Skip to content
GenAI Learn/Reinforcement Learning for LLMs
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Policy Gradients & Advantage Estimation

5 min read

You'll learn to

  • -Explain REINFORCE's core idea: weight a log-probability gradient by the outcome it led to
  • -Understand why a baseline reduces variance without changing the expected gradient, with a numeric example
  • -Compute an advantage and standardize a batch of advantages
  • -See how the same idea shows up in training reasoning models: compare each answer to the average

DQN learns Q-values and picks whichever action looks best. A policy gradient method takes a more direct path: it adjusts the probability of each action directly, based on how good the outcome was. This is the family of algorithms that everything from classic robotics RL to modern LLM reasoning training actually builds on.

An Analogy: Coaching a Shooter

A coach watches a basketball player take shots and says, do more of what worked and less of what did not. But the useful feedback is not just did it go in, it is was that better or worse than your usual shot? If a player makes 80% of their shots, a make is no news, and a miss is the interesting event. A good coach compares each shot with what to expect, and nudges the player toward the moves that beat expectation.

  • -The player's habits are the policy: the probabilities of choosing each action.
  • -How the shot went is the return.
  • -What you would normally expect is the baseline (or the value function, in the full version).
  • -How much better than expected the shot was is the advantage, and it sets how hard the habit is reinforced.

REINFORCE: Reinforce the Good, Suppress the Bad

REINFORCE's idea, stated plainly: for an action that led to a high return, increase its log-probability; for one that led to a low return, decrease it - weighted by exactly how good or bad the outcome was, relative to some baseline.

REINFORCE Loss
L = -mean((Gₜ - b) · log π(aₜ))
Weight each action's log-probability by (return - baseline), then negate for gradient descent (minimizing this loss is the same as maximizing expected return).
The REINFORCE loss, from scratch

Run it. Both actions earned a return of 10, and their log-probabilities are -0.5 and -0.2. With no baseline the loss is -(10 x -0.5 + 10 x -0.2) / 2 = 3.5. With a baseline of 2 each weight becomes 8, and the loss is -(8 x -0.5 + 8 x -0.2) / 2 = 2.8. The baseline scaled down how hard each action is pushed, and next we see why that is useful.

Why a Baseline? Variance, Not Bias

Imagine four sampled actions whose returns are 100, 102, 98 and 101. With no baseline, every action gets a weight of about 100, so all four are pushed up hard, and the useful information, that 102 beat 98 by a little, is buried under a huge shared number. Subtract a baseline of 100 and the weights become 0, 2, -2 and 1: now the better actions are pushed up, and the worse one is pushed down. The direction on average is the same, but each individual update is far less noisy.

Subtracting a well-chosen baseline (often a learned value-function estimate) does not change what the gradient points toward on average - it is still an unbiased estimate of the true policy gradient. What it does change, dramatically, is the variance of that estimate. Without a baseline, a policy trained on noisy, high-magnitude returns updates erratically; with one, updates become far more stable. This is the entire reason baselines exist.

Advantage: How Much Better Than Expected?

The advantage function formalizes "baseline" into a specific, widely-used choice: the actual return minus a learned value function's prediction. It directly answers "how much better than expected was this outcome?" - exactly the signal a policy gradient method should train on.

Advantage
A(s, a) = G - V(s)
Actual return minus the value function's own prediction for that state.
Advantage, and standardizing a batch of advantages

Run it. The advantages are 1, -1 and 4 (returns 5, 3 and 8, against a predicted value of 4). Standardizing subtracts their mean, 1.33, and divides by their standard deviation, about 2.05, giving roughly -0.16, -1.14 and +1.30. Whatever the scale of the rewards, the updates stay about the same size.

Standardizing advantages across a batch (subtract the mean, divide by the standard deviation) is standard practice - it keeps the effective learning rate stable no matter the raw scale of the reward function being used.

The Same Idea in Modern LLM Training

Training reasoning models uses the same trick. A common approach samples several answers to the same question, scores each one, and uses the average score of the group as the baseline. An answer that beats its group's average has a positive advantage and is reinforced; one that falls below it is suppressed. Nobody needs a separate learned value function, because the group provides the comparison.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Implement both from scratch: Level 57 (Policy Gradients / REINFORCE) and Level 58 (Advantage Estimation) in AI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!