Skip to content
GenAI Learn/Reinforcement Learning for LLMs
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

DPO & Test-Time Compute: Two More Ways to Get a Better Model

6 min read

You'll learn to

  • -Explain DPO as training directly on chosen-versus-rejected pairs, with no separate reward model
  • -Read the DPO loss: it rewards the policy for preferring the chosen answer MORE than the reference does
  • -Implement Best-of-N and self-consistency, and see with numbers why voting over several reasoning chains helps

The last chapter built PPO and the KL-penalized reward that RLHF optimizes. KL (Kullback-Leibler) divergence is a measure of how far one probability distribution has drifted from another, here the tuned model from the original one. PPO needs a separately trained reward model and a rollout loop that is notoriously fiddly to tune. This chapter covers two techniques that sidestep that, from two directions. DPO skips PPO altogether during training. Test-time compute improves answers at inference time, without touching training at all.

An Analogy: Learning From Comparisons

RLHF with PPO is like hiring a judge, training the judge to score essays, and then having a student write essays to please the judge. DPO is simpler: show the student two answers to the same question, say "this one is better than that one", and let them adjust directly. The comparisons carry all the information, and no judge is needed in between.

DPO: Optimizing the Same Objective, Without a Reward Model

DPO (Direct Preference Optimization) needs only pairs of (chosen, rejected) human preference examples: no reward model, no PPO rollout loop. It works because the KL-penalized RLHF objective from the last chapter has a closed-form solution, meaning an exact formula. The optimal policy can be written directly in terms of the reference model and the reward. Rearranging that relationship turns "maximize reward under a KL constraint" into a simple, direct classification loss over preference pairs.

DPO Loss
L = -log σ(β · [(log πθ(yw|x) - log πref(yw|x)) - (log πθ(yl|x) - log πref(yl|x))])
yw = chosen ("winning") completion, yl = rejected ("losing") completion. The policy is rewarded for preferring chosen over rejected MORE than the reference model already did, scaled by beta - the exact same beta that controlled the KL penalty in PPO/RLHF.

In the formula, a log-probability is the logarithm of how likely the model finds a completion, so higher means more likely, and it is negative. σ is the sigmoid function, which squashes any number into the range 0 to 1. The reference model is a frozen copy of the starting model. Now read the first example below. The policy gives the chosen answer a log-probability of -1.0 and the rejected one -2.0, a gap of 1.0 in favor of the chosen. The reference saw no difference, -1.5 for both. So the policy prefers the chosen answer by 1.0 more than the reference did, the logit is 0.1 × 1.0 = 0.1, and the loss is about 0.644, a little below the 0.693 you get when nothing has been learned.

The DPO loss, from scratch
  • -DPO needs four numbers per example: the policy's and the frozen reference model's log-probability of the chosen completion, and the same two for the rejected completion. Never a reward-model score.
  • -It is a binary classification loss in disguise: the policy is being trained to "classify" the chosen completion as the preferred one, exactly as a logistic regression classifier is trained.
  • -Because it needs no reward model, no sampling loop and no PPO-style clipping, DPO trains with ordinary supervised-style gradient descent. That is why most open-model alignment (Llama, Zephyr and others) moved to DPO instead of PPO for most preference tuning.

DPO and PPO/RLHF are not rivals with different goals. They are two ways of reaching the same KL-constrained objective. PPO reaches it step by step through sampling and a reward model. DPO reaches it in one closed-form step directly from preference pairs.

Test-Time Compute: Getting Smarter Without Retraining

Everything so far in this module improves a model by spending more compute on training it. Reasoning models such as OpenAI's o1 and DeepSeek-R1 also spend extra compute at inference time. They sample several candidate solutions to the same problem and aggregate over them, instead of trusting a single pass that always takes the likeliest next word (a greedy pass). Two of the simplest aggregation strategies:

  • -Best-of-N: sample N completions, score each with a reward model or verifier, and keep the highest-scoring one.
  • -Self-consistency: sample N independent reasoning chains, extract a final answer from each, and return whichever answer appears most often, on the theory that correct reasoning tends to converge on the same answer while flawed reasoning scatters.
Best-of-N and self-consistency, from scratch

The first result picks index 1 because its score, 0.9, is the highest. The second returns 3 because it appears three times among the five chains. The demo below puts a number on why voting helps. Suppose a solver is right 60% of the time on a single try. The question is how often the majority of several independent tries is right.

Press Run: more chains make the majority answer more reliable

Both techniques trade inference cost (N samples instead of 1) for accuracy. That is exactly why reasoning models are slower and more expensive per query than a standard chat model, and why "test-time compute scaling" is now discussed as its own lever beside training compute and model size.

AI Lab's Level 66 (Direct Preference Optimization) and Level 67 (Best-of-N & Self-Consistency) implement both of this chapter's ideas in numpy: the DPO loss over a batch, and vectorized Best-of-N and self-consistency using np.argmax and np.unique.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Implement both from scratch: Level 66 (Direct Preference Optimization) and Level 67 (Best-of-N & Self-Consistency) in AI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!