Reasoning Distillation

Reasoning distillation trains a smaller student model on a stronger teacher's reasoning. The simplest form is supervised fine-tuning on teacher-written solutions: the student learns to imitate complete chains of thought with an ordinary next-token cross-entropy loss. No reward, verifier or RL loop runs during training.

It is an alternative route to reasoning ability, alongside RL with verifiable rewards. DeepSeek-R1 distilled into Qwen and Llama models with SFT alone, and the paper reports that, for a 32B base model, distillation outperformed running large-scale RL on that model directly. Distillation is cheap when a strong teacher exists; RL is how you get past the teacher.

Separate two things a student can match. Hard distillation matches the teacher's sampled tokens: train on the text it wrote. Soft distillation matches the teacher's full probability distribution at each position, which carries information about the alternatives it considered. Soft targets need the teacher's logits and a shared vocabulary; hard targets only need text, so they work with API-only teachers.

The filter step is where verifiers return. Training on unchecked teacher output teaches the student the teacher's mistakes as well as its reasoning.

Hard (sequence-level) distillation minimizes the negative log-likelihood of teacher outputs $y$ for prompts $x$:

usually computed only on response tokens, with the prompt masked out. Soft distillation (Hinton et al.) matches temperature-softened distributions $q = \operatorname{softmax}(z_{\text{teacher}}/T)$ and $p = \operatorname{softmax}(z_{\text{student}}/T)$:

where the $T^2$ factor keeps gradient magnitudes comparable across temperatures.

Both are usually off-policy: the student trains on sequences the teacher produced, but at test time it conditions on its own earlier tokens. One early mistake puts it in states the training data never covered. On-policy distillation samples from the student and asks the teacher to score or correct those tokens, which trains on the student's own mistakes at the cost of running the teacher during training.

Key takeaway: Hard SFT on filtered teacher traces is the practical default. Its ceiling is the teacher, and its blind spot is the student's own error states.

The loss for hard distillation is ordinary SFT with the prompt masked. Labels are the input shifted by one.

1import torch
2import torch.nn.functional as F
3
4def distill_loss(logits, input_ids, prompt_len):
5    # logits: [B, L, V]; input_ids: [B, L] = prompt + teacher response
6    pred = logits[:, :-1]                      # position t predicts token t+1
7    target = input_ids[:, 1:].clone()
8    target[:, : prompt_len - 1] = -100         # ignore prompt tokens
9    return F.cross_entropy(pred.reshape(-1, pred.size(-1)),
10                           target.reshape(-1), ignore_index=-100)
11
12B, L, V = 2, 12, 100
13loss = distill_loss(torch.randn(B, L, V), torch.randint(0, V, (B, L)), prompt_len=5)

With a padded batch, use a per-example prompt length and also mask padding tokens. Evaluate on a split the teacher's solutions did not come from.

1. Conceptual: Does reasoning distillation need a reward model? (No. Hard distillation is supervised learning on teacher outputs. A verifier may filter the data, but it is not a training-time reward.)
2. Applied: Why filter teacher solutions before training? (Unfiltered traces include wrong answers; the student imitates those as faithfully as correct ones.)
3. Implementation: Why mask the prompt tokens in the loss? (The goal is to model responses given prompts, not to spend capacity predicting the prompt itself.)
4. Systems-level: When is soft distillation impossible? (When the teacher is only available as text, or when teacher and student tokenizers differ without an alignment step.)
5. Failure modes: What is exposure bias in off-policy distillation? (The student trains on teacher prefixes but generates from its own; errors compound in states the data never showed. On-policy distillation targets this.)

From memory: write the hard and soft distillation losses, explain the T² factor, and state one advantage and one cost of on-policy distillation. Check against Stage 3.

Sources and further reading

• Hinton et al., Distilling the Knowledge in a Neural Network: soft targets and temperature.
• DeepSeek-AI, DeepSeek-R1: SFT-only distillation of R1 reasoning into Qwen and Llama models, compared with RL on the smaller model.
• Agarwal et al., On-Policy Distillation of Language Models: training on student-generated sequences with teacher feedback.
• Raschka, Building Reasoning Models from the Ground Up: distills DeepSeek-R1 answers on MATH into a small Qwen3 model.