RL Fundamentals

On-Policy vs Off-Policy

The whole distinction reduces to two policies:

• Behavior policy — generates the data (chooses actions in the environment).
• Target policy — the one being learned / improved.

On-policy: behavior = target — you learn about the same policy you act with. Off-policy: behavior ≠ target — you learn about a different policy than the one that generated the data.

Canonical contrast: Q-learning vs SARSA

Both update a Q-value from a transition (s, a, r, s'); they differ in one term of the target:

• SARSA (on-policy): r + γ·Q(s', a') — uses the action a' the policy actually took next. Evaluates the policy you’re following, exploration included.
• Q-learning (off-policy): r + γ·max_{a'} Q(s', a') — uses the greedy action regardless of what behavior did. Learns the value of the optimal policy while behaving exploratorily.

That max vs actual-next-action is the single most testable illustration.

Cliff-walking intuition: SARSA learns a safer path, Q-learning the optimal-but-riskier one. SARSA’s value accounts for its own ε-greedy exploration occasionally stepping off the cliff; Q-learning assumes greedy behavior at s', so it hugs the edge.

Why off-policy is attractive

1. Data reuse — learn from data generated by old/other policies → makes replay buffers (DQN) possible. On-policy must mostly discard data after each update.
2. Sample efficiency — fewer environment interactions.
3. Learn optimal while exploring — behave exploratorily, learn about the greedy policy.
4. Learn from demonstrations / logged data (offline RL).

Why off-policy is harder

• Distribution mismatch — data comes from the behavior distribution, but you want to evaluate the target policy → naive averaging is biased.
• Correction = importance sampling — reweight each sample by π_target(a|s) / π_behavior(a|s). Fixes bias but has high variance when policies diverge.
• Instability — the “deadly triad”: off-policy learning + function approximation (neural nets)
  • bootstrapping (TD) can diverge. Hence DQN’s stabilizers (target networks, replay).

On-policy’s tradeoff (mirror image)

More stable and simpler (no importance-sampling correction; data matches the distribution you care about) but sample-inefficient — needs fresh data each update.

It’s a spectrum — the RLHF-relevant part

PPO is almost on-policy: collect a batch with the current policy, then take several gradient steps on it. The later steps are technically off-policy (policy has moved), so PPO uses an importance ratio π_new/π_old with clipping to keep the policies close and bound the off-policy error. That clipped ratio is the on/off-policy correction — and PPO is the workhorse of RLHF.

Summary

	On-policy	Off-policy
Behavior vs target	same	different
Examples	SARSA, REINFORCE, A2C, PPO*	Q-learning, DQN, DDPG, SAC
Data reuse / replay	no (mostly)	yes
Sample efficiency	lower	higher
Stability	higher	lower (deadly triad)
Correction needed	none	importance sampling

*PPO is on-policy-ish with a clipped importance ratio.

Why REINFORCE/A2C can’t reuse old trajectories: the policy gradient is an expectation over trajectories drawn from the current policy ∇J = E_{τ∼π_θ}[...]. Once θ updates, old trajectories are samples from a different distribution → using them without an importance-sampling correction gives a biased gradient. DQN sidesteps this because a Q-value obeys the Bellman equation for any transition, regardless of which policy produced it.

---

Advantage Estimation (GAE) and the Bias/Variance Tradeoff

Why the advantage

Policy gradient pushes up action probabilities weighted by how good the action was. Using the raw return Gₜ has huge variance; the fix is the advantage:

$A(s,a) = Q(s,a) - V(s)$

How much better a is than the average action from s. Subtracting V(s) is a baseline — it leaves the gradient’s expectation unchanged (unbiased) but slashes variance. The real question: how do we estimate A from samples? — and that estimation is itself a bias/variance choice.

The spectrum of estimators

You never have the true A; you build it from rewards + a learned value function V. The family is parameterized by how many real rewards you use before bootstrapping (substituting V’s estimate for the rest):

• Monte Carlo: Aₜ = Gₜ − V(sₜ) — all actual rewards to episode end. Unbiased (no reliance on V’s errors) but high variance (sum of many random rewards).
• 1-step TD: Aₜ = rₜ + γV(sₜ₊₁) − V(sₜ) — one real reward, then bootstrap. Low variance but biased (leans entirely on imperfect V).
• n-step: n real rewards then bootstrap — interpolates.

Core tension: more real rewards → less bias, more variance; more bootstrapping → less variance, more bias.

The TD residual — the building block

$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$

This is the 1-step advantage estimate. GAE is built entirely from these δ’s.

GAE = exponentially-weighted average of all n-step estimators

Instead of picking one n, GAE averages over all of them with exponentially decaying weights, controlled by λ ∈ [0, 1]:

$A_t^{GAE(\gamma,\lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l \, \delta_{t+l}$

λ is the bias/variance dial:

λ	Estimator	Bias	Variance
0	Aₜ = δₜ (1-step TD)	high	low
1	Aₜ = Σ γˡ δₜ₊ₗ = Gₜ − V(sₜ) (Monte Carlo)	none	high
0–1	smooth interpolation	—	—

Typical: λ ≈ 0.95.

λ=1 telescopes to the MC advantage: write out δₜ + γδₜ₊₁ + γ²δₜ₊₂ + …, substitute each δ = r + γV' − V, and the intermediate V terms cancel in a chain → Gₜ − V(sₜ). That telescoping is the “aha” of GAE.

Two knobs — don’t conflate them

• γ (discount): how far into the future rewards matter. Lower γ → shorter effective horizon → less variance, more bias. Primarily a problem-definition parameter with a bias/variance side effect.
• λ (GAE): how much to trust bootstrapped V vs actual multi-step rewards, given the horizon γ sets. Purely an estimator knob.

Why bootstrapping trades variance for bias

• Variance ↓: V(sₜ₊₁) is a single deterministic estimate replacing a sum of many sampled future rewards — one number instead of a noisy random sum.
• Bias ↑: during training V is wrong, so substituting it injects its error into the target. MC avoids this by using only real rewards.
• So why not always λ=1 (unbiased MC)? Its variance is so high that gradient estimates are noisy and training is slow/unstable — especially for long episodes. λ≈0.95 keeps most of the low bias while cutting variance enough to train stably.

In practice: GAE is the standard advantage estimator in A2C/PPO (typical γ=0.99, λ=0.95). The learned V (the critic) is what makes bootstrapping possible. See [[rl-fundamentals]] on-policy section — PPO consumes these GAE advantages in its clipped objective.

Related: [[optimization]], [[learning-rate]]