Policy Gradient (PPO)

Policy gradient methods optimize a parameterized policy πθ(a|s) directly by gradient ascent on expected return, instead of learning a value function and acting greedily on it as Q-learning does. Proximal Policy Optimization (PPO) (Schulman et al., 2017) is the most widely used member of the family: a simple, robust actor-critic algorithm that constrains each update so the new policy stays close to the old one. It is the default choice for continuous control benchmarks and is the RL step in classic RLHF.

1. The Policy Gradient Theorem and REINFORCE

The objective is J(θ) = Eτ~πθ[G(τ)]. The policy gradient theorem gives an estimator that needs only samples and the policy's log-probabilities, not the environment's dynamics:

∇θJ = E[ Σt ∇θ log πθ(at|st) · Ât ]

In REINFORCE, Ât is the Monte Carlo return Gt. That estimator is unbiased but very high-variance. Subtracting a baseline b(s) — usually a learned value function V(s) — leaves the expectation unchanged and cuts variance dramatically. With a learned critic, you have an actor-critic; the quantity A(s,a) = Q(s,a) − V(s) is the advantage.

Generalized Advantage Estimation (GAE) blends one-step TD errors δt = rt + γV(st+1) − V(st) with exponential weight λ: Ât = Σk (γλ)k δt+k. λ=0 is low-variance/high-bias TD; λ=1 is Monte Carlo. Values around 0.95 are common.

2. PPO's Clipped Objective

Vanilla policy gradients are sensitive to step size: one too-large update can collapse the policy, and because data is on-policy, a bad policy then collects bad data. TRPO solved this with a hard KL-divergence trust region, at the cost of second-order optimization. PPO gets most of the benefit with a first-order trick. Define the probability ratio rt(θ) = πθ(at|st) / πθold(at|st); then maximize

LCLIP = E[ min( rtÂt, clip(rt, 1−ε, 1+ε) Ât ) ]

The plots show the effect: once the ratio moves past 1+ε for a good action (or below 1−ε for a bad one), the objective goes flat and the gradient is zero, so there is no incentive to move further. The min makes it a pessimistic bound — moves that make things worse are never clipped away. That lets PPO safely run several epochs of minibatch SGD on the same rollout batch.

3. The Full Training Loop

A PPO iteration collects a fixed number of steps from (often many parallel) environments, computes GAE advantages and value targets, normalizes advantages per batch, then runs K epochs over shuffled minibatches. The combined loss adds a value-function regression term and an entropy bonus that keeps the policy from becoming deterministic too early. Policies are Gaussian (mean and log-std outputs) for continuous actions and categorical for discrete ones.

4. Python: PPO with Stable-Baselines3, and the Core Loss

For real work, use a tested implementation. Stable-Baselines3 trains PPO on a Gymnasium environment in a few lines:

# pip install stable-baselines3 gymnasium
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.evaluation import evaluate_policy

env = make_vec_env("CartPole-v1", n_envs=8, seed=0)
model = PPO("MlpPolicy", env, n_steps=256, batch_size=256, n_epochs=10,
            gamma=0.99, gae_lambda=0.95, clip_range=0.2, ent_coef=0.0,
            learning_rate=3e-4, verbose=0, seed=0)
model.learn(total_timesteps=200_000)
mean_r, std_r = evaluate_policy(model, make_vec_env("CartPole-v1", n_envs=1), n_eval_episodes=20)
print(f"eval return: {mean_r:.1f} +/- {std_r:.1f}")

The heart of PPO, written out in PyTorch, is short:

import torch

def ppo_loss(dist, old_logp, actions, adv, values, returns, clip=0.2, vf_coef=0.5, ent_coef=0.01):
    """dist: torch.distributions object from the current policy for a minibatch."""
    adv = (adv - adv.mean()) / (adv.std() + 1e-8)          # per-batch normalization
    logp = dist.log_prob(actions)
    ratio = torch.exp(logp - old_logp)                       # pi_theta / pi_old
    surr1 = ratio * adv
    surr2 = torch.clamp(ratio, 1 - clip, 1 + clip) * adv
    policy_loss = -torch.min(surr1, surr2).mean()
    value_loss = (returns - values).pow(2).mean()
    entropy = dist.entropy().mean()
    approx_kl = (old_logp - logp).mean()                      # monitor; early-stop if large
    return policy_loss + vf_coef * value_loss - ent_coef * entropy, approx_kl.detach()

5. When to Use Policy Gradients

6. Common Pitfalls

• Implementation details matter. Advantage normalization, observation normalization, orthogonal init, learning-rate annealing and value clipping all measurably affect PPO results.
• Reusing data too long. Too many epochs per batch drives the ratio outside the clip range; monitor approximate KL and clip fraction.
• Entropy collapse. The policy becomes deterministic and stops exploring; add or raise the entropy coefficient.
• Unscaled rewards make the value loss dominate; normalize returns.
• Sample inefficiency. On-policy data is thrown away after each update; budget environment steps accordingly.

7. Returns and Advantages, Worked Through

Take a three-step episode with rewards 1, 0, 2 and discount γ = 0.9. The discounted returns computed backwards are G₂ = 2, G₁ = 0 + 0.9 × 2 = 1.8, and G₀ = 1 + 0.9 × 1.8 = 2.62. REINFORCE would weight the log-probability of each action by those returns, so every action in a rewarded episode is pushed up, including the poor ones, and only averaging over many episodes sorts them out. That is the variance problem.

Now suppose a critic estimates V(s₀) = 2.0, V(s₁) = 1.5, V(s₂) = 1.0, and V = 0 after the terminal step. The TD errors are δ₂ = 2 − 1.0 = 1.0, δ₁ = 0 + 0.9 × 1.0 − 1.5 = −0.6, and δ₀ = 1 + 0.9 × 1.5 − 2.0 = 0.35. With λ = 0.95 (so γλ = 0.855), GAE gives A₂ = 1.0, A₁ = −0.6 + 0.855 × 1.0 ≈ 0.26, and A₀ = 0.35 + 0.855 × 0.26 ≈ 0.57. The action at step 1, which earned nothing immediately and landed in a state worse than the critic expected, now gets a much smaller push than under raw returns. The value-function targets are A_t + V(s_t): about 2.57, 1.76 and 2.0.

8. The Policy-Gradient Family

For continuous actions the policy network usually outputs the mean of a Gaussian, with a state-independent learned log standard deviation, and actions are clipped or squashed (SAC uses tanh with a log-probability correction) to the valid range. For discrete actions, the network outputs logits for a categorical distribution, and invalid actions can be masked by setting their logits to a large negative value.

9. Hyperparameters and Diagnostics

Reasonable PPO starting points, close to common library defaults, are: learning rate 3×10⁻⁴ with linear decay, clip range ε = 0.2, γ = 0.99, λ = 0.95, 2,048 steps per environment per rollout, 10 epochs over minibatches of 64, value-loss coefficient 0.5, entropy coefficient between 0 and 0.01, and gradient-norm clipping at 0.5. Long-horizon tasks need γ closer to 1; tasks with sparse rewards benefit from more entropy. Many implementations add a target-KL early stop that ends the epochs for a batch once the approximate KL exceeds a threshold such as 0.01 to 0.02.

Results vary a lot between random seeds, so report the mean and a confidence interval over at least five seeds, and prefer interquartile mean or similar robust aggregates (Agarwal et al., 2021) over a single best curve. Also log action statistics: a policy whose Gaussian standard deviation has shrunk to near zero has stopped exploring, and one whose actions are always at the clipping boundary probably has a scale problem.

10. Applications and Production Notes

Policy gradients are the workhorse for robot locomotion and manipulation trained in simulation and transferred with domain randomisation (see control algorithms), large-scale game agents (OpenAI Five used PPO at scale), resource allocation and scheduling where the action space is structured, and language-model alignment, where the "environment" is a reward model and a KL penalty to a reference model keeps the policy from drifting into reward hacking (see RLHF).

In production RL, the simulator is usually the largest engineering effort: it must be fast, vectorised and close enough to reality. Reward design deserves the same scrutiny as a loss function in supervised learning; agents exploit loopholes (a boat circling to collect points instead of finishing a race is a well-known example from OpenAI's 2016 post on faulty reward functions). Before deployment, evaluate the frozen policy on held-out scenarios, add safety constraints or action filters outside the learned policy, and log states and actions so failures can be replayed. Where interaction is expensive or risky, consider offline RL or imitation learning first.

Further reading: Williams, "Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning" (1992); Sutton et al., "Policy Gradient Methods for Reinforcement Learning with Function Approximation" (2000); Schulman et al., "High-Dimensional Continuous Control Using Generalized Advantage Estimation" (2015); Schulman et al., "Proximal Policy Optimization Algorithms" (2017, arXiv:1707.06347); Huang et al., "The 37 Implementation Details of Proximal Policy Optimization" (ICLR Blog Track, 2022); Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018), chapter 13.

Related Topics

• Reinforcement Learning
• Machine Learning (ML)
• Q-Learning / DQN
• RLHF
• Control Algorithms
• PyTorch