Knowledge distillation

There is information beyond the top answer.

Reveal a teacher’s softened distribution and watch a student learn its relative preferences.

Soften the distribution

Temperature changes both teacher and student probabilities. The outline bars show the teacher; filled bars show the student. A higher temperature exposes relative probability away from the top outcome.

Match the full target

The student minimizes forward KL with T² scaling. The teacher remains fixed. Every value on the graph comes from the six editable logits; the motion follows actual gradient updates.

Distinguish this from on-policy learning

This is fixed-input distribution matching, not an on-policy distillation pipeline. There are no student-generated trajectories, vocabulary-scale tensors, or claims about benchmark performance.

The equation behind the motion

q = softmax(zteacher / T), p = softmax(zstudent / T)
L = T² KL(q ∥ p)
∂L/∂zstudent = T(p − q)

An exact six-outcome illustration. It omits neural-network and sequence-level training complexity.

Read the primary source ↗