Skip to content

From the paper to the mechanism

See what changes.
Understand why.

Follow the learning signal from an example, a preference, a teacher, or a reward. These small, inspectable models expose the calculations behind the animation.

Supervision ≠ reinforcement learning

SFT learns from demonstrations. DPO learns preferences. Distillation matches a teacher. Policy-gradient methods use reward. Explore them side by side without treating them as interchangeable.

DEMONSTRATIONResponse 1The target ✓Response 3Response 4LEARN THE TARGET

Interactive mechanism

Supervised fine-tuning

Choose the demonstrated response. Watch cross-entropy move probability toward it, one gradient update at a time.

Explore the mechanism ↗
Preferred +Rejected −Preference marginlog σ

Interactive mechanism

Direct preference optimization

Pick a preferred and rejected response. Compare their log-probability ratios against a frozen reference.

Explore the mechanism ↗
TEACHERSTUDENTSoft targets · temperature

Interactive mechanism

Knowledge distillation

Reveal a teacher’s softened distribution and watch a student learn its relative preferences.

Explore the mechanism ↗
1 − ε1 + εCLIPPED SURROGATE

Interactive mechanism

PPO clipping

Change the clip range and inspect where the policy surrogate stops rewarding movement in the same direction.

Explore the mechanism ↗
A GROUP OF RESPONSESABCDEFReward relative to the group mean

Interactive mechanism

Group-relative optimization

Score six candidate responses, normalize their rewards, and trace the clipped policy update.

Explore the mechanism ↗
SAMPLE → REWARD → UPDATE23%70%40%A policy learns through experience

Interactive mechanism

Policy-gradient bandit

A seeded agent samples actions, receives their rewards, and updates its categorical policy with REINFORCE.

Explore the mechanism ↗
TEACHERREWARDmixAnneal the schedule · gate each turn

Recent paper · August 2026 revision

Teacher-to-reward guidance

Explore ATOD’s annealing and turn-level gates using editable synthetic signals.

Explore the mechanism ↗

Further reading · discovered through alphaXiv

Where the ideas are going.

Distilled Reinforcement Learning for LLM Post-training

This July 2026 paper, revised in September, investigates selective teacher signals inside reward-based learning. It is further reading, rather than an implemented training algorithm in Studio.

Discover on alphaXiv ↗Read the primary paper ↗