Policy-gradient bandit
Experience changes the next decision.
A seeded agent samples actions, receives their rewards, and updates its categorical policy with REINFORCE.
Sample a decision
Six actions have known deterministic rewards. The agent samples an action from its current policy instead of always choosing the best one. The seed makes the sequence reproducible.
Credit the sampled action
The sampled reward minus an exponential running baseline scales the log-policy gradient. The baseline used for the update is its value before observing the current reward.
Track expected reward
The trace shows the exact expected reward under the current policy, not just the latest sample. Individual updates can reduce it. This one-step environment illustrates policy gradients, not long-horizon credit assignment.
The equation behind the motion
gᵢ = −(r − baseline)(1[i = action] − pᵢ) z ← z − αg
An exact six-outcome illustration. It omits neural-network and sequence-level training complexity.
Read the primary source ↗