Reinforcement learning (RL) trains an agent to make sequences of decisions by trial and error. There is no labeled "correct action"; the agent receives a scalar reward after acting and must learn which behavior maximizes cumulative reward over time. That delayed, evaluative signal is what separates RL from supervised learning.
RL powers game-playing systems (AlphaGo, Atari DQN), robotics control, recommendation and ad bidding policies, and — via RLHF — the alignment stage of modern LLMs. The deep-dive pages below cover the three methods you are most likely to meet.
RL problems are formalized as a Markov Decision Process (S, A, P, R, γ): states, actions, transition probabilities P(s'|s,a), reward function R(s,a), and discount factor γ. A policy π(a|s) maps states to action probabilities. Two value functions summarize how good things are under a policy:
Vπ(s) = E[Gt | st=s] — expected return from state s.Qπ(s,a) = E[Gt | st=s, at=a] — expected return after taking action a in s.They satisfy the Bellman equation: Qπ(s,a) = E[r + γ Σa' π(a'|s') Qπ(s',a')]. The optimal Q-function replaces the policy average with a max: Q*(s,a) = E[r + γ maxa' Q*(s',a')]. Almost every RL algorithm is a way to estimate one of these quantities from sampled experience, or to directly improve π.
Value-based methods learn Q and act greedily on it; they suit discrete action spaces (Q-learning, DQN). Policy-gradient methods parameterize π directly and follow the gradient of expected return; they handle continuous actions and stochastic policies (REINFORCE, PPO). Actor-critic combines the two: a critic estimates values to reduce the variance of the actor's policy gradient (A2C, SAC, PPO itself).
Model-based methods learn or are given the transition dynamics and plan with them (Monte Carlo Tree Search, model predictive control, MuZero). They are far more sample-efficient when a good model is available, but model errors compound over long rollouts. See Minimax and MCTS for the planning side.
A second axis is on-policy (learn only from data generated by the current policy, e.g. PPO) vs off-policy (learn from any past data, e.g. DQN, SAC). Off-policy methods reuse experience and are more sample-efficient; on-policy methods tend to be more stable.
An agent that always picks the currently best-known action never discovers better ones. Common strategies: ε-greedy (random action with probability ε, annealed over training), softmax / Boltzmann sampling over Q-values, entropy bonuses in policy-gradient losses, Upper Confidence Bound in bandits and MCTS, and intrinsic-curiosity rewards for sparse-reward environments.
Tabular Q-learning and the Bellman update, then Deep Q-Networks with replay buffers and target networks.
REINFORCE, advantage estimation, actor-critic, and the clipped surrogate objective of Proximal Policy Optimization.
Reinforcement learning from human feedback: reward models from preference pairs, PPO with a KL penalty, and DPO.
Gymnasium (the maintained successor to OpenAI Gym) defines the standard environment interface. This runs a random policy on CartPole; every algorithm on the child pages replaces env.action_space.sample() with a learned policy.
import gymnasium as gym # pip install gymnasium
env = gym.make("CartPole-v1")
for episode in range(3):
obs, info = env.reset(seed=episode)
total, done = 0.0, False
while not done:
action = env.action_space.sample() # policy goes here
obs, reward, terminated, truncated, info = env.step(action)
total += reward
done = terminated or truncated # terminal state vs time limit
print(f"episode {episode}: return={total}")
env.close()
Distinguish terminated (true end of episode — bootstrap value is 0) from truncated (time limit hit — you should still bootstrap from the next state). Mixing them up is a classic source of biased value estimates.
Q-learning estimates Q* from single transitions without a model of the environment. Suppose Q(s, right) = 2.0, the agent takes right, receives r = 1, and lands in s' where the best current estimate is maxa' Q(s', a') = 5.0. With γ = 0.9 the target is 1 + 0.9 × 5.0 = 5.5. The temporal-difference error is 5.5 − 2.0 = 3.5, and with learning rate α = 0.1 the new value is 2.0 + 0.1 × 3.5 = 2.35. The estimate moves a small step toward a target that itself depends on other estimates, which is called bootstrapping. Repeated over many visits, values propagate backward from rewarding states.
The discount factor sets the horizon: rewards k steps ahead are weighted by γk, so γ = 0.99 gives an effective horizon of roughly 1 / (1 − γ) = 100 steps. Monte Carlo methods instead wait until the episode ends and use the actual return, which is unbiased but high-variance; n-step returns and GAE (generalized advantage estimation) interpolate between the two.
| Hyperparameter | Role and guidance |
|---|---|
| Discount γ | 0.99 is a common default; lower it for short-horizon tasks to reduce variance. |
| Learning rate | Often around 3e-4 with Adam for deep RL; too high causes value divergence or policy collapse. |
| Exploration | ε-greedy schedule for DQN (decay from 1.0 to a small floor); entropy bonus for policy gradients. |
| Replay buffer and target network | Off-policy value methods; buffer size and target update rate trade stability against freshness. |
| Rollout length, batch size, clip range | PPO-specific; the clip range (commonly 0.2) limits how far each update moves the policy. |
| Random seeds | Results vary widely across seeds; report several runs, not one. |
The reward is the specification, and agents optimize exactly what it says. Sparse rewards (1 for reaching a goal, 0 otherwise) are faithful but hard to learn from; shaped rewards that give partial credit learn faster but invite reward hacking, where the agent finds an unintended way to score, such as circling to collect a bonus repeatedly instead of finishing a race. Potential-based shaping (Ng, Harada and Russell, 1999) is a form of shaping that provably preserves the optimal policy. The same issue appears in RLHF, where a policy can exploit weaknesses in a learned reward model, which is why a KL penalty to the reference model is used.
Evaluate with the exploration noise turned off, over many episodes and several training seeds, and report mean return with a confidence interval. Watch learning curves, not only final scores, and test on environment variations the agent did not train on to check that it has not overfit a single layout or simulator.
Further reading. Sutton and Barto (2018), Reinforcement Learning: An Introduction, 2nd edition; Watkins and Dayan (1992), "Q-learning"; Mnih et al. (2015), "Human-level control through deep reinforcement learning", Nature; Schulman et al. (2017), "Proximal Policy Optimization Algorithms", arXiv:1707.06347; Silver et al. (2016), "Mastering the game of Go with deep neural networks and tree search"; Henderson et al. (2018), "Deep Reinforcement Learning that Matters"; Levine et al. (2020), "Offline Reinforcement Learning: Tutorial, Review, and Perspectives".