Skip to content
GenAI Learn/Reinforcement Learning for LLMs
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Reinforcement Learning Foundations

6 min read

You'll learn to

  • -Name the parts of reinforcement learning (agent, environment, state, action, reward, policy) and map them onto an LLM
  • -Compute a discounted return by hand and in code, and say what the discount factor gamma controls
  • -Explain what a Q-value is and why learning from a guess about the future (bootstrapping) is useful
  • -Compute a one-step TD target, including the terminal case

Reinforcement learning (RL) is learning by trial and reward instead of by examples with answers. You met it briefly in Phase 1. It matters here for a specific reason: the reasoning-focused LLMs released since 2024, such as OpenAI's o1 and DeepSeek-R1, are trained with reinforcement learning on top of an already-trained base model. So RL is now part of how the most capable models are made, and the next three chapters build up to it.

An Analogy: Training a Puppy

A puppy tries things, such as sitting, jumping or rolling over, and sometimes gets a treat. It is never told the right move for each situation. Over many tries it works out which habits earn treats. The treat often arrives only at the end of a whole sequence of actions, so the puppy must also work out which earlier actions deserved the credit.

  • -The puppy is the agent: the learner that chooses actions.
  • -The living room is the environment: everything the agent interacts with and gets feedback from.
  • -Sitting or rolling over are actions.
  • -The treat, or its absence, is the reward.
  • -The puppy's habits for what to do in each situation are its policy.

The Vocabulary in One Place

  • -State: what the agent can see right now.
  • -Action: a choice the agent makes in that state.
  • -Reward: a number the environment sends back, saying how good that step was.
  • -Policy: the agent's rule for choosing actions. Training improves the policy.
  • -Episode: one complete run from start to finish, like one game.
  • -Return: the total reward collected over an episode, or from a given moment onward.

Now map this onto a language model. The prompt is the state. Generating a response is the action. A reward model, a separate network trained to score responses, or a rule-based checker (for example, one that verifies a math answer) returns the reward. The model's way of choosing words is its policy.

Discounted Return: Why Later Rewards Count Less

An agent should care about its whole future, but a reward far away should count for less than one right now, just as $10 today is worth more than $10 in a year. We express that with a discount factor gamma, a number between 0 and 1. A reward one step away is multiplied by gamma, two steps away by gamma twice, and so on. Add everything up and you get the discounted return, the quantity nearly every RL algorithm tries to maximize.

Discounted Return
Gₜ = Σₖ γᵏ rₜ₊ₖ
Add up every future reward, but multiply the reward k steps ahead by gamma to the power k.

A worked case: rewards of 0, 0 and 10, with gamma = 0.9. The return is 0 + 0.9 x 0 + 0.81 x 10 = 8.1. The same 10 delivered immediately would count in full. With gamma = 0.5 the delayed 10 shrinks to 2.5, and with gamma = 0 it vanishes entirely, which is why a gamma near 0 makes an agent short-sighted. Run it to check.

Discounted return, from scratch

Use the demo below to feel what gamma does. Move the reward further into the future, then lower gamma, and watch how little of it the agent still cares about today.

How Much Is a Future Reward Worth Today?

An agent cares about its whole future, but a reward far away counts for less than one right now. The discount factor gamma sets how fast that value fades. Choose when the reward arrives and how steeply it fades.

0.90
2
10
0123456789

Bar height: how much a reward at that step counts today (gamma to the power of the step). Blue marks the step your reward arrives.

Worth today

8.10

of 10 at the moment it arrives

Counts for

81%

gamma^2

A reward of 10 that arrives 2 steps away is worth 8.10 today at gamma 0.90. Roughly the next 10 steps matter to an agent with this gamma.

Q-Values and Bootstrapping: Learning Before the Game Ends

A Q-value, written Q(s, a), is the agent's estimate of the total discounted reward it will collect if it takes action a in state s and plays well from then on. Q stands for quality: how good is this action, here? If the agent knows Q for every action in a state, it can simply pick the best one.

The difficulty is learning Q. Waiting for a whole episode to finish before learning anything is slow, and sometimes impossible, because some tasks never end. The trick is bootstrapping: update a guess using another guess, instead of waiting for the final answer. Think of a GPS during a drive.

  • -After 10 minutes of driving, the app says 25 minutes remain. You can already say the trip will take about 35, without waiting to arrive.
  • -The 10 minutes you actually drove is the real reward r.
  • -The app's estimate for the rest of the trip is the discounted best Q-value of the next state, gamma times the max over actions.
  • -Your updated estimate is the TD target (TD stands for temporal difference): r + gamma x max Q(next state).
  • -Arriving is the terminal case. Nothing is left to estimate, so the target is just r.
DQN's TD Target
target = r + γ · maxₐ′ Q(s′, a′)
This step's real reward, plus a discounted guess at the best achievable value from the next state. If the episode just ended, the target is only r.

A Deep Q-Network (DQN) is Q-learning in which a neural network stores the Q-values, and training nudges its predictions toward this target. Take a step that earns reward 1.0, landing in a state whose best action has Q = 5.0, with gamma = 0.9. The target is 1.0 + 0.9 x 5.0 = 5.5. If that step ended the episode, the target is simply 1.0.

The TD target that Q-learning trains toward

AI Lab's Level 55 (Discounted Returns) and Level 56 (Q-Values & the Bellman Update) implement both of these directly: the discounted return formula, and the exact TD-target computation every DQN update trains toward.

Most LLM training does not use DQN. It uses policy gradient methods, which the next two chapters cover. But the ideas from this chapter keep returning: a reward for each answer, a discounted or total return, and a learned estimate of how good a situation is, which is what the advantage in the next chapter is built from.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Implement both from scratch: Level 55 (Discounted Returns) and Level 56 (Q-Values & the Bellman Update) in AI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!