Group-relative optimization
The group supplies the comparison.
Score six candidate responses, normalize their rewards, and trace the clipped policy update.
Compare within one group
Each response receives an editable reward. The mean and population standard deviation produce group-relative advantages. Equal rewards produce zero advantages and no learning signal here.
Watch the relative update
Higher-than-average responses have positive advantages. The same clipped surrogate used in PPO controls the local objective. The reference remains the initial policy across this miniature optimization run.
This is the actor mechanism
The preset omits token-level aggregation, KL regularization, neural networks, and new rollouts. It demonstrates one fixed group rather than reproducing the full DeepSeekMath training procedure.
The equation behind the motion
Aᵢ = (rewardᵢ − mean(rewards)) / std(rewards) L = −mean[min(rᵢAᵢ, clip(rᵢ,1−ε,1+ε)Aᵢ)]
An exact six-outcome illustration. It omits neural-network and sequence-level training complexity.
Read the primary source ↗