Reinforcement Learning
Reinforcement Learning
Reinforcement learning trains an agent through trial and reward rather than labels. This lesson covers the agent-environment loop, exploration and Q-learning.
advanced · 16 min read
Lesson content
Supervised learning learns from examples with known correct answers. Unsupervised learning finds structure with no answers at all.
Reinforcement learning, usually shortened to RL, is a third kind entirely. There is no answer key, but there is feedback. An agent takes actions, and the world responds with a reward that says roughly how good or bad each one turned out to be.
The difference from supervised learning is subtle and it changes everything. A label tells you what you should have done. A reward only tells you how well what you did went, and often only much later. The agent has to work out for itself which of its many earlier choices deserve the credit.
Definition
Reinforcement learning
Training an agent to make good decisions by letting it try actions in an environment and learn from the rewards (or penalties) those actions produce.
Formal definition
A machine learning approach in which an agent learns a policy for choosing actions by interacting with an environment over time, using scalar reward signals to reinforce actions that lead to better outcomes.
Training a dog with treats
You cannot hand a dog a labeled dataset of correct behaviors.
Instead you let it try things, and you reward the ones you want repeated: a treat for sitting on command, nothing for ignoring you. Over many repetitions the dog learns which actions, in which situations, tend to lead to a treat.
It never sees an explicit rule. It builds a sense of what to do from the pattern of rewards alone.
The analogy also carries the classic failure. Reward a dog for bringing you a stick and it may learn to break sticks into pieces so it can bring you several. It did what was rewarded, not what you meant, and reinforcement learning agents do exactly the same thing.
Example
A robot finding its way across a grid
Picture a robot placed on a small grid of squares. Most squares are open floor. One is a wall it cannot pass through, and one is the goal.
On every square the robot can move up, down, left or right. It gets a small penalty for each move, a bigger penalty for bumping into the wall, and a large reward for reaching the goal.
Nobody tells the robot the shortest path. It has to discover one by trial and error, and the small per-move penalty is what makes it care about shortness at all: without it, a path of forty moves and a path of four would earn the same reward.
This grid runs through the rest of the lesson, because every idea in RL is easier to see on a world you can draw.
- Agent
The learner or decision-maker, the robot in the grid-world example.
Everything in reinforcement learning is described from the agent's point of view: what it observes, what it does, what it earns.
- Environment
Everything outside the agent that it interacts with, the grid, the wall and the goal.
The environment decides what happens next and what reward to give, in response to whatever action the agent takes.
- State
A snapshot of the situation the agent is currently in, here, which square of the grid it is standing on.
The agent's decisions are based on its current state. The same action can lead to different outcomes from different states.
- Action
One of the choices available to the agent at a given moment, up, down, left, or right.
Learning, in RL, ultimately means learning which action to pick in which state.
- Reward
A number the environment hands back after an action, saying how good or bad that action was.
Reward is the only feedback the agent gets. No label ever tells it the "correct" action directly, only whether things went well, so whatever you reward is what it will pursue.
- Policy
The agent's strategy: a rule for which action to take in each state.
A policy is what the agent is ultimately trying to improve, the whole point of training is ending up with a policy that earns high reward.
- Episode
One full run from a starting state to an ending point, like the robot going from its start square to the goal (or getting stuck too long).
The agent usually improves gradually across many episodes, not within a single one.
The reinforcement learning loop
Observe the current state
The agent notices which situation it is in, for the robot, which square of the grid it is on.
Choose an action
Based on its current policy (and, as covered below, a bit of experimentation), the agent picks up, down, left, or right.
The environment responds
The robot moves (or bumps the wall) and the environment returns a reward for that action along with the new state.
Update what it knows
The agent uses the reward and new state to adjust its sense of how good that action was from that state.
Repeat until the episode ends
This continues, square by square, until the robot reaches the goal (or the episode is cut off), then a new episode begins.
Why does the agent sometimes pick an action it does not think is best?
Because acting only on its current best guess forever means it can never discover something better that it simply has not tried yet.
This is the exploration versus exploitation trade-off. Exploitation means taking the action the agent currently believes is best. Exploration means occasionally trying something else, on purpose, only to find out whether it is actually better.
Early on, the robot's beliefs are nearly random, so trusting them completely would lock in whichever path it happened to stumble on first.
A common approach, called epsilon-greedy, has the agent exploit most of the time and explore at random a small fraction of the time, often 10 percent. That is enough to keep discovering without abandoning what already works, and a later lesson is devoted to getting the balance right.
Definition
Q-value
A number estimating how good it is to take one specific action from one specific state, in terms of future reward.
Formal definition
The expected total future reward of taking a given action in a given state and following the current policy thereafter, denoted Q(state, action).
| State (square) | Up | Down | Left | Right |
|---|---|---|---|---|
| A1 | 0.2 | 0.6 | 0.1 | 0.4 |
| A2 | 0.3 | 0.8 | 0.2 | 0.5 |
| B1 | 0.1 | 0.2 | 0.5 | 0.9 |
Each cell is a Q-value, how good the robot currently believes that action is from that square. From B1, "Right" has the highest Q-value (0.9), so a purely exploiting robot would move right from there.
Why store all this as a lookup table instead of a single formula?
A Q-table is exactly that lookup: one row per state, one column per action, and a Q-value in each cell answering "how good is this action, from here?"
For a small grid with a handful of squares and four possible moves, a table like this is simple, exact and easy to inspect. You can read off the best action from any square by scanning its row for the highest number, and that row-by-row reading is the policy.
A 5 by 5 grid with four moves is a table of 100 cells. That is small enough to print, which makes it the right place to learn the idea before the states become too numerous to list.
The math
Every time the agent tries an action and sees what reward and next state follow, it nudges that action's Q-value a little toward the new evidence, partly trusting its old estimate, partly updating toward the reward just received plus the best it now expects from the next state. Repeated many times, across many episodes, these small nudges converge toward genuinely useful estimates of how good each action really is.
new Q-value = old Q-value + learning rate × (reward + best next Q-value − old Q-value)
- old Q-value
- The agent's current estimate for this state-action pair, before this update.
- reward
- What the environment actually handed back for taking this action.
- best next Q-value
- The agent's current best estimate for the best action available from the new state it landed in.
- learning rate
- How much weight the new evidence gets versus the old estimate, the same idea as the learning rate in training a neural network.
This update rule is called Q-learning, and repeating it across many episodes is how the Q-table above gets filled in from scratch.
Two related ideas are worth knowing by name, without their full derivations.
SARSA is a close cousin that updates using the action the agent actually takes next, exploration included, rather than the best possible one. That makes it more cautious: near a cliff edge it learns the safer route, because it accounts for the occasional random step.
Deep Q-Networks replace the table with a neural network. That becomes necessary once there are far too many states to list, such as every possible arrangement of pixels on a game screen, and it is the subject of its own lesson later in this path.
# A minimal Q-learning update for one (state, action, reward, next_state) step.# q_table maps state -> {action: q_value}q_table = { "A1": {"up": 0.2, "down": 0.6, "left": 0.1, "right": 0.4}, "A2": {"up": 0.3, "down": 0.8, "left": 0.2, "right": 0.5},}learning_rate = 0.1state, action, reward, next_state = "A1", "down", -1, "A2"old_value = q_table[state][action]best_next_value = max(q_table[next_state].values())q_table[state][action] = old_value + learning_rate * (reward + best_next_value - old_value)print(q_table[state][action])q_table = {
"A1": {"up": 0.2, "down": 0.6, "left": 0.1, "right": 0.4},
"A2": {"up": 0.3, "down": 0.8, "left": 0.2, "right": 0.5},The Q-table is just a nested lookup: for each state, a Q-value for every possible action, exactly the dataset shown above, as plain Python.
state, action, reward, next_state = "A1", "down", -1, "A2"This single step imagines the robot was on A1, moved down, received a reward of -1 (a small step penalty) and landed on A2.
best_next_value = max(q_table[next_state].values())best_next_value looks up the best Q-value available from wherever the agent landed, the "best next Q-value" from the formula above.
q_table[state][action] = old_value + learning_rate * (reward + best_next_value - old_value)This line is the Q-learning update rule itself, applied directly: blend the old estimate with the new evidence, weighted by the learning rate.
When to use
- There is no labeled dataset of "correct" decisions, but you can define a clear reward signal for good and bad outcomes
- The problem naturally involves a sequence of decisions where earlier actions affect later options, like navigation or game-playing
- You can let an agent practice repeatedly, in simulation or in a safe real environment, to gather enough experience to learn from
When to avoid
- Mistakes during learning are costly or unsafe in the real world and cannot be simulated first
- A supervised approach already has plenty of labeled examples of correct decisions, that is usually simpler and more data-efficient
- The number of possible states is small and well understood and a hand-written rule would work just as well without any learning at all
Expecting the agent to be handed a correct answer for each situation, the way supervised learning is.
Reinforcement learning only ever gets a reward number after acting, the agent has to work out good behavior indirectly, through trial and error, not from labeled examples.
Rewarding the agent only at the very end (reaching the goal) and nothing along the way.
Very sparse rewards make learning painfully slow, since the agent gets almost no feedback for most of its actions, designing rewards that give some signal along the way often helps considerably.
Cutting exploration to zero as soon as the agent finds any policy that works.
An early policy that works is not necessarily the best one, keeping some exploration alive lets the agent keep discovering better actions instead of settling permanently for a mediocre habit.
Key takeaway
- Reinforcement learning trains an agent through reward and consequence, not labeled examples.
- The loop is: observe a state, choose an action, receive a reward and a new state, update and repeat.
- Exploration vs. exploitation is a real, ongoing trade-off, occasionally trying a non-best action is how the agent discovers better ones.
- A Q-table stores how good each action looks from each state. Q-learning is the update rule that fills it in from experience, while SARSA and DQN extend the same core idea.
Finished this lesson?
Guest progress is saved on this device.