Reinforcement Learning: Learning From Reward
You'll learn to
- -Define reinforcement learning and its core components: agent, environment, action, reward
- -Write the discounted return formula and explain what the discount factor controls
- -Understand the exploration-exploitation tradeoff
Reinforcement learning is a fundamentally different setup from supervised or unsupervised learning. Instead of learning from a fixed dataset, an agent takes actions in an environment and receives rewards or penalties based on the outcomes, gradually learning a strategy, called a policy, that maximizes cumulative reward over time.
The Four Core Pieces
- -The agent is the decision-maker being trained: a game-playing program, a robot, a recommendation system.
- -The environment is everything the agent interacts with and receives feedback from.
- -An action is a choice the agent makes at a given moment.
- -A reward is a signal telling the agent how good or bad the outcome of its action was.
A classic example is training an agent to play a video game. The agent tries actions, sometimes stumbling into a high score by accident, and gradually learns which sequences of actions tend to lead to reward. No one hand-labels "the correct move" at each step. The agent has to work that out from delayed, sparse reward signals.
What the Agent Actually Maximizes: Discounted Return
A reward at any single step is rarely the whole story. An action might sacrifice a small reward now for a much bigger one several steps later. RL formalizes this with the "return": the sum of all future rewards from this point on, with a discount factor that controls how much the agent should care about reward far in the future versus reward right now.
- -A discount factor γ (gamma) close to 0 makes the agent short-sighted, caring almost only about the immediate next reward.
- -A γ close to 1 makes the agent far-sighted, weighing rewards many steps in the future almost as heavily as immediate ones.
- -γ is always strictly less than 1 for an infinite horizon. Without it, the sum could grow without bound and "maximize the return" would be meaningless.
Exploration vs. Exploitation
A central tension in RL is whether the agent should exploit the best strategy it has found so far, or explore new actions that might turn out even better. Too much exploitation and the agent gets stuck in a mediocre local strategy. Too much exploration and it never capitalizes on what it has already learned. Balancing this tradeoff is one of the central open problems in the field.
Reinforcement learning is the technique behind game-playing systems that beat human champions, and it is also a core ingredient in how modern large language models are fine-tuned to be more helpful and less harmful, a technique called reinforcement learning from human feedback, or RLHF, which you will meet again in Phase 2.
A team wants to use reinforcement learning to optimize a recommendation system for long-term engagement, not just the next click. What would you want to clarify about the reward signal before agreeing this is a good fit?
"I would just have it maximize clicks at every step, since more clicks is always better."
"I would want to separate the reward from the metric it is actually meant to serve. Reinforcement learning optimizes exactly what it is rewarded for, so if the reward only reflects the next click, the agent becomes short-sighted, similar to using a discount factor near zero, and it can learn to game immediate engagement in ways that hurt long-term retention. I would want a reward signal and discount factor that genuinely reflect delayed, longer-term outcomes, and I would want to understand how much exploration the system can safely do in production before trusting its policy with real users."
What are the four core components of a reinforcement learning setup?