LLM_log #039: Policy Gradient Baselines — From REINFORCE to GRPO

LLM_log #039: Policy Gradient Baselines — From REINFORCE to GRPO

Policy Gradient Baselines — From REINFORCE to GRPO

Highlights: Subtracting a simple 0.5 baseline cuts the gradient’s standard deviation from 0.325 to 0.108 without biasing the update, and GRPO gets its baseline for free from a group of rollouts. In CMU 11-768 AI Agents (Fall 2026), Daniel Fried explains why reinforcement learning for language agents requires relative reward baselines rather than raw trajectory returns. We walk through the mathematical proof of zero-bias baselines, analyze variance reduction in a two-armed bandit simulation, and trace the evolution from PPO to critic-free methods like GRPO and DrGRPO.

  • Zero gradient on failure: Standard REINFORCE leaves policy parameters unchanged on zero-reward trajectories, completely preventing error correction on failed attempts.
  • Zero baseline bias: Action-independent baselines conditioned strictly on history introduce zero gradient bias because total action probabilities sum to one.
  • Bandit variance reduction: Centering binary rewards with a 0.5 baseline reduces gradient estimator standard deviation from 0.325 to 0.108 while maintaining the identical expected update of 0.1875.
  • Critic-free GRPO: Group-Relative Policy Optimization eliminates auxiliary value networks by calculating empirical baseline means and standard deviations across rollouts sampled per prompt.
  • DrGRPO corrections: DrGRPO removes group standard deviation scaling and replaces variable-length division with a constant budget to prevent task difficulty distortion and verbosity drift.

Tutorial Overview:

  1. Why Rewards Must Be Relative
  2. Baselines and the Advantage Function
  3. A Minimal Two-Armed Bandit Simulation
  4. Intermediate Rewards: The Reward-to-Go Formulation
  5. Group-Relative Policy Optimization (GRPO)
  6. DrGRPO: Fixing Pathologies in Standard GRPO
  7. Failure Modes, PPO Comparison, and Unification
  8. Summary

1. Why Rewards Must Be Relative

In short
– Raw trajectory rewards yield zero gradient on failures, completely preventing error correction.
– Baselines convert raw returns into relative advantages to penalize mistakes appropriately.
– Rare failures on easy tasks require strong negative updates against expected returns.

1.1 Limits of raw trajectory rewards

Standard policy gradient methods fail when updates depend purely on unshifted, absolute reward values.

  • REINFORCE (a policy gradient algorithm scaling updates by unshifted rewards): leaves parameters unchanged on zero-reward failures.
  • $R = 0$ update elimination: produces zero gradient in REINFORCE and ReST, ignoring fatal mistakes.
  • $R = 1$ equal upweighting: boosts every intermediate step equally regardless of individual action quality.
  • 95% success baseline (a reference value subtracted from rewards): makes rare 5% failures yield massive negative updates.
  • Advantage (a metric measuring return relative to expected return): scales policy gradients (parameter updates proportional to advantages) effectively.
Trajectory Guesses and feedback Reward Policy update
$\tau_1$ (Success) 8 (higher), 12 (lower), 10 (higher), 11 (correct) $R = 1$ Upweights all steps equally
$\tau_3$ (Failure) 16 (lower), 8 (higher), 9 (higher), 10 (timeout) $R = 0$ Zero gradient; ignores mistakes

Trajectory reward breakdown showing that rare mistakes on easy tasks must yield large corrective gradient penalties.

Fig 1. Rare mistakes on easy tasks require strong negative updates against expected baseline rewards.

Shifting rewards creates relative signals so agents learn which specific choices caused failure. Relative scaling ensures gradient steps reflect task difficulty rather than arbitrary reward scales.


2. Baselines and the Advantage Function

In short
– Advantage evaluates whether an action outperforms the policy’s average expected return.
– History-conditioned baselines reduce gradient variance without introducing estimator bias.
– Baseline designs determine core algorithm families including REINFORCE, PPO, and GRPO.

2.1 Formalizing Advantages in Guessing Games

Fried formalizes advantages in a number guessing game where agents discover hidden numbers through sequential guesses.

  • Advantage metric: Advantage (A metric measuring how much better a specific action or trajectory is compared to the policy’s average expected return.) evaluates action quality: $A^\pi(h, a) = Q^\pi(h, a) – V^\pi(h)$.
  • Action-value definition: $Q^\pi(h, a)$ computes expected return after choosing action $a$ under autoregressive policy $\pi$.
  • State-value definition: $V^\pi(h)$ computes expected return when sampling all subsequent actions directly from policy $\pi$.
  • Fried’s guessing setup: Guesses 8 (“higher”) and 12 (“lower”) leave target candidates 9, 10, and 11.
  • 1 versus ~2/3 return: Optimal policies achieve 1 expected return; random valid policies achieve roughly 2/3 return.
  • 10 versus 11 advantage: Guess 10 yields $A > 0$; guess 11 leaves two remaining candidates ($A < 0$).

Lecture slide 27: Advantages

Fig 2. Advantage functions quantify action quality relative to the baseline state value.

Centering action returns around state values highlights genuinely superior choices. It prevents penalizing good actions taken in difficult states.

2.2 Policy Gradients with Baseline Functions

Empirical advantage estimators subtract history-dependent baselines from sampled trajectory rewards to approximate intractable value expectations.

  • Empirical advantage formula: Empirical advantage $\hat{A}_t = R(\tau) – b(h_t)$ approximates unobservable theoretical advantages during training.
  • Baseline role: Baseline (A reference value subtracted from trajectory rewards to evaluate whether an outcome was better or worse than expected without biasing gradient updates.) approximates state return $V^\pi(h_t)$.
  • REINFORCE update: REINFORCE (A basic policy gradient algorithm that scales updates by the raw, unshifted trajectory reward, failing to actively penalize zero-reward failures.) scales policy gradients by raw reward $R(\tau)$:
    [\widehat{\nabla_\theta J} = R(\tau) \sum_t \nabla_\theta \log \pi_\theta(a_t \mid h_t)]
  • Policy gradient update: Policy gradient (A method that optimizes a policy’s parameters by adjusting action probabilities proportionally to observed rewards or advantages.) updates parameters using advantages $\hat{A}t$:
    [\widehat{\nabla
    \theta J} = \sum_t \left( R(\tau) – b(h_t) \right) \nabla_\theta \log \pi_\theta(a_t \mid h_t)]
  • Baseline design options: Implementations use constant offsets, running reward means, parametric value models, or group rollouts.
Algorithm Baseline Method Critic Model Advantage Target
REINFORCE None ($b=0$) No critic Raw reward $R(\tau)$
PPO (Proximal Policy Optimization) (An actor-critic reinforcement learning algorithm that relies on an auxiliary parametric value model to estimate baselines and advantages.) Parametric Value model (Critic) (A dedicated neural network trained alongside a policy to predict expected future returns from a given state or history.) Separate value network Shifted reward $R(\tau) – V_\phi(h_t)$
GRPO (Group-Relative Policy Optimization) (An algorithm that computes advantages by comparing multiple rollouts generated from the same prompt, eliminating the need for a separate critic model.) Group rollout mean No critic network Group-normalized return

Comparison of different baseline choices and their resulting policy gradient algorithm families.

Fig 3. Baseline configurations establish distinct reinforcement learning algorithm families.

Lecture slide 28: Policy gradients with advantage estimates

Fig 4. Centering trajectory rewards around baselines yields practical policy gradient advantage estimators.

Subtracting baselines recenters rewards to reduce sample variance without altering expected gradients. This stabilizes policy updates.

2.3 Proof of Zero Gradient Bias

History-conditioned baselines introduce zero estimation bias while significantly reducing variance across sampled trajectory updates.

  1. Express the expected value of the baseline-subtracted policy gradient estimator:
    [\mathbb{E}\left[\widehat{\nabla_\theta J}\right] = \nabla_\theta J – \mathbb{E}\left[ \sum_t b(h_t) \nabla_\theta \log \pi_\theta(a_t \mid h_t) \right]]
  2. Expand trajectory expectations using linearity into an outer expectation over histories and inner over actions:
    [\mathbb{E}{a_t \sim \pi\theta}\left[ b(h_t) \nabla_\theta \log \pi_\theta(a_t \mid h_t) \right] = \sum_{a_t} \pi_\theta(a_t \mid h_t) b(h_t) \nabla_\theta \log \pi_\theta(a_t \mid h_t)]
  3. Factor the action-independent baseline $b(h_t)$ outside the action summation:
    [b(h_t) \sum_{a_t} \pi_\theta(a_t \mid h_t) \nabla_\theta \log \pi_\theta(a_t \mid h_t)]
  4. Apply the log-derivative identity $\nabla_\theta \log \pi_\theta(a \mid h) = \frac{\nabla_\theta \pi_\theta(a \mid h)}{\pi_\theta(a \mid h)}$ to eliminate policy probabilities:
    [b(h_t) \sum_{a_t} \nabla_\theta \pi_\theta(a_t \mid h_t)]
  5. Interchange summation and gradient operators to take the gradient of total probability:
    [b(h_t) \nabla_\theta \sum_{a_t} \pi_\theta(a_t \mid h_t) = b(h_t) \nabla_\theta (1)]
  6. Differentiate constant 1 to zero, confirming the baseline term vanishes:
    [\mathbb{E}\left[\widehat{\nabla_\theta J}\right] = \nabla_\theta J]
  • History dependence requirement: Baselines must depend strictly on history $h_t$ rather than current action $a_t$.
  • 1 total probability: Probabilities across the complete action space sum to exactly 1.
  • 0 constant gradient: Gradients of constant values equal zero, eliminating the baseline term in expectation.
  • Bias and variance: Baselines reduce empirical trajectory variance without introducing statistical bias into gradient calculations.

Step-by-step mathematical proof that baselines independent of action choice yield zero gradient bias.

Fig 5. Action-independent baselines vanish in expectation because total action probability sums to one.

Baselines conditioned strictly on history avoid action correlations. This ensures policy gradients remain mathematically unbiased.


3. A Minimal Two-Armed Bandit Simulation

In short
– Baselines reduce gradient variance without shifting the expected policy gradient update.
– Centering rewards turns zero-reward failures into active corrective gradient signals.
– Scalar baselines cannot isolate credit among sequential actions in multi-step trajectories.

3.1 Bandit Parameterization and Setup

Daniel Fried examines baseline mechanics using a two-armed bandit (a simplified single-decision reinforcement learning setting).

  • Single parameter $\theta$: scalar parameter controls action probabilities through a standard sigmoid squashing function.
  • -1.1 initial value: starting parameter establishes a strong policy bias toward the failing action.
  • 0.25 probability on A: guess A yields reward 1 but samples only one-quarter of runs.
  • 0.75 probability on B: guess B yields reward 0 and dominates three-quarters of runs.
  • Positive parameter shift: policy gradient (a method optimizing policy parameters via rewards) pushes $\theta$ upward.

[p = \pi_\theta(\text{guess } A) = \sigma(\theta) = \frac{1}{1 + e^{-\theta}}]

[1 – p = \pi_\theta(\text{guess } B) = 1 – \sigma(\theta) = 1 – p]

Action Probability Reward Log Derivative
Guess A 0.25 1 +0.75
Guess B 0.75 0 -0.25
  1. Pass the current scalar parameter $\theta$ into the sigmoid activation function.
  2. Compute the resulting action probabilities $p$ and $1 – p$.
  3. Sample an action choice according to the computed probability distribution.
  4. Observe the numerical reward returned by the selected bandit arm.

Action distribution for the two-armed bandit showing heavy bias toward the zero-reward action.

Fig 6. Action B dominates samples while Action A rarely yields reward.

Negative parameters squash sigmoid outputs below 0.5, concentrating probability on Action B. Shifting $\theta$ positive increases rewards.

3.2 Gradient Calculation Without Baseline

Calculating gradients without baselines produces noisy estimates that frequently stall policy parameter updates.

  • +0.75 score derivative: derivative of log-probability evaluates to $1 – p$ for guess A.
  • -0.25 score derivative: derivative of log-probability evaluates to $-p$ for guess B.
  • +0.75 sampled update: sampling Action A scales positive derivative by full reward 1.
  • 0.00 sampled update: sampling Action B zeroes out the gradient estimate entirely.
  • 0.1875 analytical expectation: true expected gradient averages both sampled outcomes weighted by probability.
  • 75% stalled rate: three out of four samples generate zero parameter updates.

[\frac{\partial}{\partial \theta} \log \pi_\theta(\text{guess } A) = 1 – p = 1 – 0.25 = +0.75]

[\frac{\partial}{\partial \theta} \log \pi_\theta(\text{guess } B) = -p = -0.25]

[\hat{g}A = R \times \frac{\partial}{\partial \theta} \log \pi\theta(\text{guess } A) = 1 \times (+0.75) = +0.75]

[\hat{g}B = R \times \frac{\partial}{\partial \theta} \log \pi\theta(\text{guess } B) = 0 \times (-0.25) = 0]

[\mathbb{E}[\hat{g}] = p \cdot (+0.75) + (1 – p) \cdot (0) = 0.25 \times 0.75 + 0.75 \times 0 = 0.1875]

  1. Calculate log-probability derivatives with respect to the scalar parameter $\theta$.
  2. Multiply the sampled action derivative by the raw observed trajectory reward.
  3. Update parameter $\theta$ in proportion to the resulting stochastic gradient estimate.

Multiplying by zero reward zeroes out gradients, ignoring valuable failure data. This creates high variance across updates.

3.3 Centering Rewards With Baselines

Subtracting a baseline (a reference value subtracted from rewards) turns negative outcomes into active learning signals.

  • 0.5 constant baseline: subtracting 0.5 centers binary rewards symmetrically around the reference anchor.
  • +0.5 positive advantage: Action A generates positive advantage (a metric measuring relative outcome quality).
  • -0.5 negative advantage: Action B generates negative advantage indicating worse than expected performance.
  • +0.375 baselined gradient: Action A multiplies positive advantage by positive derivative, boosting $\theta$.
  • +0.125 corrective gradient: Action B multiplies negative advantage by negative derivative, boosting $\theta$.

[\hat{A} = R – b]

[\hat{A}_A = 1 – 0.5 = +0.5]

[\hat{A}_B = 0 – 0.5 = -0.5]

[\hat{g}A = \hat{A}_A \times \frac{\partial}{\partial \theta} \log \pi\theta(\text{guess } A) = (+0.5) \times (+0.75) = +0.375]

[\hat{g}B = \hat{A}_B \times \frac{\partial}{\partial \theta} \log \pi\theta(\text{guess } B) = (-0.5) \times (-0.25) = +0.125]

[\mathbb{E}[\hat{g}] = 0.25 \times (+0.375) + 0.75 \times (+0.125) = 0.09375 + 0.09375 = 0.1875]

Action Reward Gradient ($b=0$) Gradient ($b=0.5$)
Guess A 1 +0.750 +0.375
Guess B 0 0.000 +0.125
Expectation – 0.1875 0.1875

Lecture slide 31: Simple example: Adding a baseline

Fig 7. Centering rewards with a baseline triggers positive updates even from failures.

Two negative signs cancel when multiplying negative advantage by negative slope. Thus, failing actions actively push parameters correctly.

3.4 Variance Reduction and Convergence Paths

Baseline centering drastically reduces estimator variance while preserving identical mathematical expected gradient updates.

  • 0.325 -> 0.108 standard deviation: subtracting constant baseline cuts gradient estimator dispersion by two-thirds.
  • 0.1875 identical expectation: analytical mean remains precisely 0.1875 regardless of baseline choice.
  • {0, 0.75} discrete spread: un-baselined estimator alternates between zero updates and large spikes.
  • {0.125, 0.375} discrete spread: baselined updates remain close to the analytical expected gradient.
  • 20-step empirical trial: un-baselined trajectories oscillate sharply compared to smooth baselined update trajectories.
  1. Initialize the policy parameter at scalar value $\theta = -1.1$.
  2. Draw twenty consecutive action samples from the parameterized distribution.
  3. Evaluate stochastic gradient updates with and without scalar baseline subtraction.
  4. Track cumulative parameter trajectories against the analytical expectation of 0.1875.

Lecture slide 33: Simple example: noise in the gradient estimate

Fig 8. Baselined gradient trajectories exhibit substantially less noise across sample draws.

Comparison of gradient estimator standard deviations with and without a constant baseline.

Fig 9. A constant baseline slashes standard deviation from 0.325 down to 0.108.

Subtracting a constant does not bias expectation because expected baseline derivatives integrate to zero. Reduced variance accelerates convergence.

3.5 Multi-Step Credit Assignment Limits

Scalar baselines benchmark overall trajectory performance but fail to resolve sequential action credit assignment.

  • Uniform scalar advantage: every intermediate token receives the identical advantage value $\hat{A}$.
  • Credit assignment limitations: scalar baselines cannot isolate which intermediate actions enabled success.
  • 8, 12, 10, 11 guess chain: number guessing game shows identical credit applied across sequential attempts.
  • +0.5 uniform credit: all four intermediate guesses receive identical reinforcement despite differing utility.
  • Value model (Critic) requirement: state-dependent baselines $V^\pi(h)$ require training dedicated neural networks.
Feature Scalar Baseline Critic Model
Computes Advantage Trajectory-level State-level
Resolves Credit No Yes
Training Cost Zero High

Capabilities and structural shortcomings of scalar baseline adjustments.

Fig 10. Scalar baselines stabilize updates but fail to resolve multi-step credit assignment.

Uniform rewards credit all intermediate actions equally regardless of their individual contribution. Sequential environments require state-dependent evaluation.


4. Intermediate Rewards: The Reward-to-Go Formulation

In short
– Reward-to-go weights each action using only subsequent rewards accumulated later in the trajectory.
– Dropping uninfluenced past feedback slashes gradient variance without introducing any update bias.
– Tasks with purely terminal payouts evaluate all intermediate steps using identical trajectory returns.

4.1 Filtering Future Returns with Reward-to-Go

Intermediate feedback environments benefit from reward-to-go (accumulating only subsequent rewards, filtering past payouts).

Feature Trajectory Return $R(\tau)$ Reward-to-Go $R_t$
Past tool reward (+0.2) Included in update Discarded from update
Terminal reward (+1) Included in update Included in update
Estimator variance High gradient noise Reduced update variance
  • $R_t$ formula: $R_t = \sum_{t’=t+1}^{T} r_{t’}$ sums rewards accumulated strictly after time step $t$.
  • +0.2 tool reward: dropped from $a_t$ update because earlier feedback is unaffected by current choices.
  • +1 terminal reward: retained in $R_t$ because downstream final payouts directly depend on action $a_t$.
  • 8, 12, 10, 11 guesses: receive identical $R_t = R(\tau)$ returns when Number Search payouts occur terminally.
  • Variance reduction: filtering out historical rewards produces cleaner policy gradient updates across multi-step sequences.

Lecture slide 35: What if we have intermediate rewards?

Fig 11. Reward-to-go lowers gradient variance by discarding past rewards from policy updates.

Past rewards cannot depend on the current action. Dropping them removes unnecessary noise without introducing estimator bias.


5. Group-Relative Policy Optimization (GRPO)

In short
– GRPO discards dedicated critic networks by sampling trajectory groups directly per prompt.
– Trajectory advantages normalize against empirical within-group return averages and standard deviations.
– Independent prompt grouping prevents difficulty distortions across heterogeneous training minibatches.

5.1 Eliminating Critics with Group Statistics

GRPO (Group-Relative Policy Optimization) [An algorithm that computes advantages by comparing multiple rollouts generated from the same prompt, eliminating the need for a separate critic model.] samples rollout groups to form a baseline [A reference value subtracted from trajectory rewards to evaluate whether an outcome was better or worse than expected without biasing gradient updates.].

Approach Baseline Source Advantage Scaling Value Network Needed
PPO [An actor-critic reinforcement learning algorithm that relies on an auxiliary parametric value model to estimate baselines and advantages.] Value model (Critic) [A dedicated neural network trained alongside a policy to predict expected future returns from a given state or history.] Critic return error Yes
GRPO Empirical group mean Group standard deviation No
DrGRPO [A variant of GRPO that removes group standard deviation scaling and token-length division to prevent task difficulty distortion and length drift.] Empirical group mean Unscaled mean centering No
  • 1 critic network eliminated: avoids training an auxiliary value network to compute baselines.
  • Shao et al. DeepSeekMath: introduced GRPO to eliminate critic models during mathematical reasoning training.
  • $G$ candidate rollouts: sampled directly from the policy model for each input prompt.
  • Empirical group mean: serves as baseline return $\bar{R} = \frac{1}{G}\sum_{j=1}^G R(\tau_j)$ across rollouts.

Standard policy optimization requires baselines to reduce return variance across rollouts. Group empirical averages eliminate extra parameter overhead while preserving mathematically unbiased baseline expectations.

5.2 Calculating Trajectory Advantages in Practice

Empirical advantage [A metric measuring how much better a specific action or trajectory is compared to the policy’s average expected return.] estimates normalize trajectory returns within each sampled group.

  1. Sample $G$ complete candidate rollouts from the policy for the input prompt.
  2. Score each trajectory outcome to obtain empirical scalar reward $R(\tau_j)$.
  3. Calculate empirical baseline mean $\bar{R} = \frac{1}{G} \sum_{j=1}^G R(\tau_j)$.
  4. Compute standard deviation $\sigma_R = \operatorname{std}{R(\tau_1), \dots, R(\tau_G)}$.
  5. Center rewards and divide by standard deviation: $\hat{A}_j = \frac{R(\tau_j) – \bar{R}}{\sigma_R}$.
  • Number guessing game: an agent discovers a hidden target number through sequential guesses.
  • 4 sampled trajectories: rollouts $\tau_1$ and $\tau_2$ succeed while $\tau_3$ and $\tau_4$ timeout.
  • $R = 1$ for $\tau_1$ and $\tau_2$: guesses 8, 12, 10, 11 receive positive success rewards.
  • $R = 0$ for $\tau_3$ and $\tau_4$: guesses 16, 8, 9, 10 or 1, 2, 3, 4 fail.
  • $\bar{R} = 0.5$ and $\sigma_R = 0.5$: centered rewards equal $+0.5$ for successes and $-0.5$ for failures.
  • $\hat{A} = +1$ versus $-1$: successes receive positive advantage while failures receive negative updates.
  • Degenerate group [A rollout sample where all trajectories achieve identical rewards (all successes or all failures), resulting in zero advantage and stalled gradient updates.]: yields zero variance and produces no gradient update.

Lecture slide 37: Group-relative advantage estimates

Fig 12. Group rollouts compute empirical return statistics to evaluate relative trajectory advantage without critics.

Group baselines remain mathematically unbiased because baselines depend strictly on prompts rather than individual actions. DrGRPO omits standard deviation scaling to prevent distorting optimization gradients across varying prompt difficulties.

5.3 Batching Multi-Prompt Rollouts

Minibatches contain multiple distinct task instances, each generating an isolated group of repeated rollout trajectories.

  • 8 rollouts per group: practical setting noted by Fried for repeated prompt sampling.
  • Isolated prompt statistics: each prompt $x_i$ evaluates independent mean $\bar{R}_i$ and standard deviation $\sigma_i$.
  • 0.25 mean for hard tasks: prompt $x_2$ yields lower empirical return due to difficulty.
  • 0.75 mean for easy tasks: simpler prompts achieve higher success rates across rollouts.
  • Credit assignment [The challenge of determining which individual intermediate actions contributed to an overall outcome across a multi-step sequence.]: operates across multi-step sequential interactions within each task instance.
  • Two-armed bandit [A simplified reinforcement learning environment where an agent makes a single discrete choice between two actions rather than a sequential chain of steps.]: contrasts with sequential guessing games by lacking multi-step states.

Group mean rewards computed separately per prompt instance to normalize varying problem difficulties.

Fig 13. Isolating group statistics per prompt normalizes advantages across varied problem difficulty levels.

Computing baselines across full minibatches would unfairly penalize harder prompts with lower empirical rewards. Isolated group statistics ensure calibrated advantages tailored strictly to specific task complexity.

5.4 Policy Gradient Objectives and Updates

Policy gradient [A method that optimizes a policy’s parameters by adjusting action probabilities proportionally to observed rewards or advantages.] objectives rescale token log-probabilities using normalized group advantages.

Standard policy gradient loss:
[\mathcal{L}{\text{PG}}(\theta) = -\sum_i R(\tau_i) \sum_t \log \pi\theta(a_{i,t} \mid h_{i,t})]

GRPO policy gradient loss:
[\mathcal{L}{\text{GRPO}}(\theta) = -\sum_i \frac{1}{G} \sum{j=1}^G \hat{A}{ij} \sum_t \log \pi\theta(a_{ij,t} \mid h_{ij,t})]

  • REINFORCE [A basic policy gradient algorithm that scales updates by the raw, unshifted trajectory reward, failing to actively penalize zero-reward failures.]: scales updates using unshifted raw returns rather than group advantages.
  • Reward-to-go [A formulation where each action is weighted solely by rewards accumulated after that step occurs, filtering out irrelevant past rewards.]: weights individual tokens solely by subsequent rewards in trajectories.
  • 1 advantage estimate $\hat{A}_{ij}$: rescales all action log-probabilities across rollout $j$ in prompt $i$.
  • 3 omitted PPO mechanisms: excludes clipping mechanics, importance-sampling ratios, and KL-divergence penalties from DeepSeekMath.
  • Empirical sample weights: adjust policy parameters directly from observed group performance.

Isolating empirical group advantages provides a practical mechanism to optimize policies without value networks. Reference model penalties and clipping prevent policy updates from taking excessively large gradient steps.


6. DrGRPO: Fixing Pathologies in Standard GRPO

In short
– DrGRPO removes standard deviation scaling to preserve true task difficulty awareness across groups.
– Constant budget division replaces response length normalization to prevent drift on wrong answers.
– Every token in a rollout shares the same unscaled, centered advantage coefficient.

6.1 Removing Standard Deviation and Length Normalization

DrGRPO modifies standard GRPO advantage scaling and token aggregation to eliminate two severe policy training biases.

Component Standard GRPO DrGRPO
Advantage denominator Group standard deviation $\sigma_R$ None (raw centered $R_j – \bar{R}$)
Token normalizer Variable response length $ y_j
Instance weighting Distorts easy and hard tasks Preserves natural task difficulty
Length behavior Drifts toward long wrong answers Penalizes wrong answers equally
  • GRPO (Group-Relative Policy Optimization): an algorithm computing advantages by comparing multiple prompt rollouts without a separate critic.
  • DrGRPO: a variant of GRPO removing group standard deviation scaling and token-length division.
  • Advantage (relative outcome quality): a metric measuring how much better a specific action or trajectory is.
  • Group standard deviation removal: unnormalized advantage $R_j – \bar{R}$ prevents easy or hard instances from inflating gradients.
  • Response length division removal: replaces token count $\frac{1}{|y_j|} \sum_t (\cdot)$ with fixed maximum budget constant $\frac{1}{C} \sum_t (\cdot)$.
  • Length drift prevention: fixed budget scaling stops policies from generating needlessly verbose incorrect answers.

Architectural differences in advantage scaling and token aggregation between standard GRPO and DrGRPO.

Fig 14. DrGRPO discards group standard deviation and variable length division to stabilize reinforcement training.

Small standard deviations on trivial or near-impossible prompts artificially inflate gradient updates. Variable length division inadvertently rewards verbosity on failing rollouts.

6.2 Per-Token Advantage Sharing Across Trajectories

Every token in a rollout receives an identical gradient weight based on its centered trajectory return.

  • Unnormalized centered advantage: calculated directly for trajectory $\tau_j$ as $\hat{A}_j = R(\tau_j) – \bar{R}$ without standard deviation division.
  • Uniform token weighting: assigns every generated token in rollout $\tau_j$ the identical coefficient $\frac{\hat{A}_j}{C}$.
  • +0.5 / C token weight: assigned to actions guess(8) and guess(11) when trajectory $\tau_1$ scores advantage $\hat{A}_1 = +0.5$.
  • -0.5 / C token weight: assigned to actions guess(16) and guess(10) when trajectory $\tau_3$ scores advantage $\hat{A}_3 = -0.5$.
  • Aggregated DrGRPO loss: optimizes token probabilities across $G$ sampled rollouts where active tokens have mask $m_{j,t} = 1$:

[
\mathcal{L}{\text{DrGRPO}} = -\frac{1}{G} \sum{j=1}^G \frac{1}{C} \sum_{t: m_{j,t}=1} \hat{A}j \log \pi\theta(u_{j,t} \mid u_{j,<t})
]

Lecture slide 40: Token coefficients in DrGRPO

Fig 15. Every token in a trajectory shares a constant-divided centered advantage during policy updates.

Uniform token coefficients ensure credit spreads evenly across all sequential guesses in the game. This prevents later reasoning tokens from receiving disproportionate gradient penalties during optimization.


7. Failure Modes, PPO Comparison, and Unification

In short
– Degenerate groups yield identical rewards and zero gradients, requiring active intervention.
– GRPO discards critic networks by calculating baselines directly across sampled groups.
– Policy algorithms share an action-weighted gradient objective addressing exposure and data mismatch.

7.1 Degenerate groups and mitigations

Optimization stalls when rollouts in a sample return identical rewards without any empirical variance.

  • [0, 0, 0, 0] rewards: all trajectories fail when tasks exceed the policy’s current capabilities.
  • [1, 1, 1, 1] rewards: all trajectories succeed when tasks offer no challenge to the agent.
  • 0 advantage: degenerate groups (rollouts with identical rewards) produce zero gradients under GRPO.
  • Number guessing stall: guesses like 8, 12, and 10 all fail, yielding no comparative signal.
  • Curriculum adaptation: adjusts task difficulty dynamically to restore reward spread across rollouts.
  • Expert rollouts: injects demonstration trajectories directly into sample pools to guarantee non-zero success.
  • Reward shaping: supplies intermediate feedback signals to score incremental multi-step guessing progress.

Intervention strategies designed to generate reward spread when all sampled trajectories fail.

Fig 16. Interventions restore empirical variance when every sampled rollout in a group fails.

Zero advantage (a metric measuring how much better a trajectory performs versus average) halts updates entirely. Policy gradient (adjusting action probabilities proportionally to rewards) requires non-zero advantage variance to guide weight adjustments.

7.2 PPO versus GRPO architecture

GRPO replaces parametric value baselines with empirical advantage estimates computed across sampled output groups.

  • Value model (Critic): dedicated neural network predicting expected future returns in PPO actor-critic setups.
  • GAE computation: calculates advantage estimates using state baseline predictions from the auxiliary critic model.
  • Shared backbones: standard PPO architectures couple policy and value heads, inflating memory overhead.
  • G candidates: GRPO samples completions for prompt query $q$ without auxiliary value networks.
  • Empirical baseline: baseline (a reference subtracted to evaluate relative performance) derives directly from completion returns.
  • Memory savings: GRPO eliminates parameter synchronization costs between policy and dedicated critic networks.
Feature Standard PPO GRPO
Critic model Required separate value network None; uses group statistics
Advantage calculation Generalized Advantage Estimation Group-relative score centering
Memory footprint High (policy, value, reference) Lower (policy and reference)

Lecture slide 42: GRPO vs PPO

Fig 17. GRPO replaces critic networks with empirical group statistics computed across candidate trajectories.

Separate critic networks require massive compute and encounter severe training instabilities during fine-tuning. Group estimation circumvents parametric baseline fitting by measuring performance relative to peer rollouts.

7.3 Unified policy action weighting

Policy gradient formulations and self-training methods unify under a shared action-weighted optimization objective.

[\widehat{\nabla_\theta J} = \sum_t w_t \, \nabla_\theta \log \pi_\theta(a_t \mid h_t)]

  • REINFORCE weighting: scales steps by raw episode return without actively penalizing complete failures.
  • ReST-EM weighting: filters trajectories with binary indicators, executing supervised updates strictly on successes.
  • GRPO weighting: weights actions by group-relative advantages normalized across multiple completions.
  • DrGRPO weighting: removes standard deviation divisors to prevent distorted task difficulty scaling.
  • Two-armed bandit: simplified reinforcement learning decisions illustrate how relative weighting shifts action selections.
  • Reward-to-go: credit assignment (attributing multi-step outcomes to specific choices) isolates rewards after each step.

Unified view showing how foundational policy gradient and self-training methods assign action weights.

Fig 18. Policy gradient and self-training objectives differ strictly through their chosen action weights.

The unified objective reweights token log-probabilities using trajectory outcomes to guide parameter updates. The specific choice of weight $w_t$ governs credit assignment and update magnitude across tokens.

Algorithm Action weight ($w_t$) Baseline mechanism
REINFORCE Raw trajectory return $R$ None
ReST-EM Binary success indicator $\mathbb{I}_{\text{success}}$ Filter threshold
PPO Critic-based GAE $\hat{A}_t$ Parametric value model
GRPO Group-normalized return $\tilde{A}_i$ Empirical group mean

Lecture slide 44: Comparing the methods we’ve seen so far

Fig 19. Reinforcement methods systematically vary their advantage baselines and sample filtering mechanisms.

Different reinforcement learning objectives select distinct weighting functions to optimize policy updates. The selected weighting mechanism dictates computational complexity and gradient variance during training.

7.4 Core reinforcement learning benefits

Sampling trajectories directly from active policies and weighting by relative reward resolves structural training bottlenecks.

  • Task mismatch: directly optimizes expected reward so any successful sequence receives positive reinforcement.
  • Data mismatch: relative baselines extract constructive gradient signals even from sub-optimal rollout batches.
  • Exposure bias: on-policy sampling forces models to encounter mistakes and learn recovery steps.
  • Target discovery: number guessing policies learn error corrections when guesses overshoot target values.

On-policy exploration prevents distribution drift by evaluating states the current policy actually generates. Reweighting trajectories ensures agents systematically favor high-reward actions across sequential decisions.


8. Summary

  • Degenerate groups: identical group rewards yield zero advantage, requiring curriculum, demonstrations, or shaped rewards.
  • GRPO efficiency: empirical group-relative statistics eliminate memory overhead and training instability of value models.
  • Unified gradients: foundational algorithms share a policy gradient template varying solely in action weights.
  • Distribution alignment: on-policy trajectory sampling resolves task mismatch, data mismatch, and autoregressive exposure bias.

Source

Lecture: Reinforcement Learning Basics by Daniel Fried, Carnegie Mellon University.
Course: CMU 11-768 AI Agents (Fall 2026), Lecture 9, second half. The first half is covered in LLM_log #038.
Video: YouTube · Slides: PDF · Course page: cmu-agents.com

Further reading referenced in the lecture:


dataHacker.rs — LLM_log #039: Policy Gradient Baselines — From REINFORCE to GRPO. By Vladimir Matic.