Clipping changes the incentive.
Change the clip range and inspect where the policy surrogate stops rewarding movement in the same direction.
Freeze the rollout policy
The initial categorical distribution is the old policy. Each of the six outcomes is treated as one fixed rollout observation with a supplied advantage.
Inspect both sides of clipping
Positive advantages stop gaining objective value above 1+epsilon; negative advantages stop gaining below 1−epsilon. The highlighted count marks terms whose direct surrogate derivative is clipped.
Clipping is not a hard probability constraint
Softmax couples the outcomes, so a clipped term’s probability can still change due to other terms. This isolates the actor surrogate without a value function, entropy bonus, or fresh rollout collection.
The equation behind the motion
rᵢ = pᵢ / pold,ᵢ L = −mean[min(rᵢAᵢ, clip(rᵢ,1−ε,1+ε)Aᵢ)]
An exact six-outcome illustration. It omits neural-network and sequence-level training complexity.
Read the primary source