Policy Gradient
Skip the Q-values — directly optimise action probabilities. Trajectories with high return get more likely; bad ones get less likely
01 Core Concept (Plain English)
DQN learns to estimate the "value" Q(s,a) of every action, then picks the highest. That works well for discrete actions, but breaks down for continuous action spaces (e.g. robot joint angles — you can't take an argmax over infinite choices).
Policy gradient takes a different approach: directly output action probabilities π_θ(a|s) from a neural network, then use gradient ascent to make good trajectories more likely and bad ones less likely.
The key formula is the policy gradient theorem: ∇J(θ) = E[G_t · ∇log π_θ(a_t|s_t)]. Intuition: large G → large gradient weight → the actions that led there get boosted.
Value-based vs Policy-based
Value-based (DQN)
Learns Q(s,a); policy = argmax Q. Requires discrete actions; can't handle continuous action spaces directly.
Policy-based (Policy Gradient)
Directly learns π(a|s); handles continuous actions; can learn stochastic policies for exploration. But high variance, slower to converge.
REINFORCE Algorithm
θ ← θ + α · G_t · ∇log π_θ(a_t|s_t)
G_t = discounted cumulative return from step t
Run the current policy π_θ in the environment; record (s₀,a₀,r₁,s₁,a₁,r₂,...) for the full episode.
G_t = r_t + γr_{t+1} + γ²r_{t+2} + ... — earlier steps look further into the future.
Large G_t → large weight on ∇log π → that action's probability increases more.
Step 1: What is a policy π(a|s)?
Step 2: Policy gradient theorem — the REINFORCE formula
Step 3: Variance problem and Baseline
REINFORCE gradient estimates have high variance. Subtracting a baseline reduces variance without changing the expected gradient:
Step 4: Full REINFORCE training (GridWorld)
Train on a 4×4 grid and watch the policy evolve from random to goal-directed:
REINFORCE's limitation: must wait for the full trajectory before updating (Monte Carlo), leading to high variance and poor sample efficiency. The fix: Actor-Critic (use a Critic to estimate V(s) as the baseline) → A2C/A3C → PPO.
02 Code
Try changing LR, GAMMA, and EPISODES to see how learning rate and discount factor affect convergence speed.
03 Deep Dive
Deriving the policy gradient theorem
J(θ) = E_τ[R(τ)]. Taking the gradient and applying the log-derivative trick: ∇p(τ) = p(τ)·∇log p(τ). The trajectory probability p(τ) = ∏π_θ(a_t|s_t)·P(s_{t+1}|s_t,a_t); in log space the environment transition terms vanish, leaving only policy terms. This is why environment dynamics don't need to be differentiable — we only need to be able to sample from them.
The progression
REINFORCE (1992)
Williams. Monte Carlo returns, high variance, no baseline.
Actor-Critic
Actor outputs policy; Critic estimates V(s) as baseline. TD error (Advantage) replaces G_t — lower variance, online updates.
PPO (2017)
Actor-Critic + clipped objective to prevent too-large policy updates. Currently the most widely used policy gradient algorithm.
Policy Gradient vs Q-Learning
Prefer Policy Gradient
Continuous or high-dimensional action space; need stochastic policy for exploration; policy has structural constraints.
Prefer Q-Learning / DQN
Small discrete action space; sample efficiency matters (can use Replay Buffer); off-policy learning needed.