GRPO and Verifiable Rewards

Group Relative Policy Optimization (GRPO) is a policy-gradient method introduced in DeepSeekMath and used to train DeepSeek-R1. For each prompt, it samples a group of responses, scores them, and uses each response's reward relative to the group as its advantage. It keeps PPO's clipped objective but drops the learned value model.

Paired with a programmatic checker, this is reinforcement learning with verifiable rewards (RLVR): the reward comes from checking the final answer or running tests, not from a reward model trained on human preferences. GRPO is the optimizer; the verifier is the reward. Either can be swapped independently.

Ask the policy the same question six times. Three answers pass the verifier and three fail. Without a value model, you still need a baseline: was a reward of 1 good or just typical? GRPO uses the group itself. Responses above the group mean are pushed up; those below are pushed down. If all six pass, or all six fail, every advantage is zero and that prompt contributes no learning signal.

This is why prompt difficulty matters. Prompts that the policy always solves, or never solves, are wasted compute for GRPO.

For prompt $q$, sample $G$ outputs $o_1,\dots,o_G$ from the old policy and score them with rewards $r_i$. With outcome rewards, every token in output $i$ shares the advantage

With the per-token ratio $\rho_{i,t} = \pi_\theta(o_{i,t}\mid q, o_{i,<t}) / \pi_{\theta_{\text{old}}}(o_{i,t}\mid q, o_{i,<t})$, the objective maximized is

DeepSeekMath adds the KL term to the loss, rather than folding it into the reward as many PPO setups do, and estimates it per token with $\frac{\pi_{\text{ref}}}{\pi_\theta} - \log\frac{\pi_{\text{ref}}}{\pi_\theta} - 1$, which is never negative.

Key takeaway: GRPO trades a learned critic for extra samples per prompt. The group mean is the baseline, so a reward only teaches the model something when responses to the same prompt disagree.

What the tracking metrics tell you

During training, watch reward, response length, entropy and the fraction of groups with zero advantage. Rising reward with collapsing entropy can mean the policy stopped exploring. Rising length is not evidence of better reasoning by itself. A growing zero-advantage fraction means the data has become too easy or too hard for the current policy. Later work, such as Dr. GRPO, also questions whether dividing by response length and reward standard deviation introduces biases, so treat the normalization choices as design decisions to test.

Compute group-relative advantages and the clipped token-level loss. Shapes: logp_new, logp_old, logp_ref and mask are [G, T]; rewards is [G].

1import torch
2
3def grpo_loss(logp_new, logp_old, logp_ref, rewards, mask, eps=0.2, beta=0.04):
4    std = rewards.std(unbiased=False)
5    adv = (rewards - rewards.mean()) / (std + 1e-6)          # [G]; zeros if all rewards equal
6    ratio = torch.exp(logp_new - logp_old)                     # [G, T]
7    surrogate = torch.minimum(ratio * adv[:, None],
8                              ratio.clamp(1 - eps, 1 + eps) * adv[:, None])
9    log_r = logp_ref - logp_new
10    kl = torch.exp(log_r) - log_r - 1                          # per-token, >= 0
11    per_token = surrogate - beta * kl
12    per_seq = (per_token * mask).sum(1) / mask.sum(1).clamp(min=1)
13    return -per_seq.mean()                                     # minimize the negative
14
15G, T = 6, 5
16logp = torch.randn(G, T)
17loss = grpo_loss(logp, logp.detach(), logp.detach(),
18                 rewards=torch.tensor([1., 0., 1., 0., 0., 1.]),
19                 mask=torch.ones(G, T))

In a real loop, logp_old comes from the policy that generated the samples, logp_ref from a frozen reference model, and gradients flow only through logp_new.

1. Conceptual: What replaces the value model in GRPO? (The mean reward of a group of responses to the same prompt acts as the baseline.)
2. Applied: Why does a prompt where all responses pass give no gradient? (All rewards equal the mean, so every advantage is zero.)
3. Implementation: Where does GRPO put the KL term? (In the loss, as a per-token estimate against a frozen reference policy, rather than inside the reward.)
4. Systems-level: What does GRPO cost instead of a critic? (G rollouts per prompt, which dominates compute for long reasoning responses.)
5. Failure modes: Why can a verifiable reward still be hacked? (The verifier checks only what it tests: a lenient answer parser, a format reward, or weak unit tests can all be satisfied without solving the task.)

From memory: write the group-relative advantage, explain why the clip only limits the incentive to move the ratio, and name two metrics to watch besides reward. Check against Stage 3.

Sources and further reading

• Shao et al., DeepSeekMath: introduces GRPO, its group baseline, and the KL estimator.
• DeepSeek-AI, DeepSeek-R1: GRPO with rule-based accuracy and format rewards at scale.
• Lambert et al., Tülu 3: names and evaluates reinforcement learning with verifiable rewards.
• Liu et al., Understanding R1-Zero-Like Training: analyses length and standard-deviation normalization in GRPO (Dr. GRPO).