PPO Explained: Clipping and Stable Policy Updates

01 Core Concept (Plain English)

Policy gradient (REINFORCE) has a fundamental problem: update step size is hard to control. Too small and training crawls; too large and the policy collapses — one bad update destroys the policy, the next rollout is garbage, and training spirals downward.

TRPO fixed this mathematically with a KL-divergence constraint but required expensive second-order optimisation. PPO achieves the same effect with a single line: clip the policy ratio r(θ) = π_new/π_old to [1-ε, 1+ε], so gradient updates beyond that range are simply blocked.

PPO also uses an Actor-Critic architecture: the Actor outputs the policy, the Critic estimates V(s) to compute the Advantage — "how much better is this action than average?"

From Policy Gradient to PPO

Policy Gradient (REINFORCE)

Wait for full trajectory, update with G_t. High variance, each sample used once, slow.

TRPO (2015)

KL-divergence constraint keeps new policy close to old. Stable but requires second-order optimisation — expensive.

PPO (2017)

Replace KL constraint with a clip operation. First-order gradients, reuse each batch multiple times, simple to implement and scale.

PPO Clipped Objective

            L_CLIP(θ) = E[ min( r(θ)·A, clip(r(θ), 1-ε, 1+ε)·A ) ]

            r(θ) = π_θ(a|s) / π_θ_old(a|s),  ε = 0.2 by default

Step 1: Policy ratio r(θ) = π_new / π_old

Step 2: Clipped objective — the circuit breaker

Step 3: Advantage estimation (GAE)

GAE uses λ to interpolate between TD and Monte Carlo, trading off bias and variance:

Step 4: Full PPO training (Actor-Critic + Clip)

Train on GridWorld with PPO and compare stability against plain REINFORCE:

Why is PPO everywhere? ChatGPT's RLHF (Reinforcement Learning from Human Feedback) uses PPO: a reward model is trained on human preference data, then PPO optimises the language model policy against it. PPO's stability and simplicity make it the default choice for RL fine-tuning of large models.

02 Code

Try changing EPSILON (clip range) and PPO_EPOCHS (update passes per batch) to observe how they affect training stability.

03 Deep Dive

Why does clipping work?

When A > 0 (good action), once r exceeds 1+ε the gradient becomes zero — the action is already sufficiently boosted. When A < 0 (bad action), once r drops below 1-ε the gradient also becomes zero — we don't over-penalise. This acts as an implicit KL penalty without computing KL divergence explicitly.

Full PPO loss function

            L = L_CLIP − c₁·L_VF + c₂·H[π]

            L_VF: Critic MSE loss; H[π]: policy entropy (encourages exploration); c₁≈0.5, c₂≈0.01

Key hyperparameters

ε (clip range)

Typically 0.1–0.3. Smaller = more conservative. ε=0.2 is the OpenAI paper default and works well for most tasks.

GAE λ

Typically 0.9–0.99. Smaller = more TD-like (lower variance, higher bias). λ=1 equals Monte Carlo.

PPO Epochs

How many passes over each batch, typically 3–10. Too many lets the new policy drift far from the old one, breaking the clip guarantee.

PPO vs other algorithms

vs REINFORCE

PPO reuses each batch multiple times (near off-policy), more stable updates, better sample efficiency.

vs TRPO

Similar performance but PPO only needs first-order gradients — simpler code, easy to parallelise.

vs SAC (continuous)

SAC often outperforms PPO on continuous control. PPO dominates for discrete actions and LLM fine-tuning (RLHF).