LLMs
Temperature, Top-k & Top-p Sampling
Watch temperature reshape probabilities and distinguish that transformation from top-k, top-p, and sampling.
Jump to a section
Why this matters
Temperature changes the shape of the probability distribution used to select an output token. Dividing logits by a lower positive temperature concentrates probability; raising temperature makes the distribution flatter.
Temperature, truncation, and sampling are separate operations. Top-k or top-p chooses which candidates remain eligible, then a token is drawn from the renormalized distribution. A temperature of zero is usually handled as a special greedy-decoding setting.
Build the mental model
Temperature scales the logits before softmax. Low temperature () sharpens the distribution toward the most likely token (approaching greedy); high temperature () flattens it, giving rarer tokens a real chance: more creative, more error-prone. Top-k and top-p then truncate: top-k keeps the k highest-probability tokens; top-p keeps the smallest set whose probabilities sum to at least p, adapting how many candidates survive to how confident the model is.
Temperature reshapes a token distribution
Compare the probabilities at low and high temperature. The largest logit retains its rank while the distribution changes.
The visual loads as you reach this section.
Work through the math
Given logits , temperature rescales before softmax:
is the raw distribution; concentrates all mass on the argmax (greedy); flattens toward uniform.
Top-k keeps the set of the highest-probability tokens, zeros the rest, and renormalizes. Top-p (nucleus) keeps the smallest set such that
then renormalizes and samples from it. The key difference: top-k uses a fixed candidate count regardless of confidence; top-p uses a dynamic count: few candidates when the model is sure, many when it's uncertain. Beam search isn't sampling at all: it maintains the highest-probability partial sequences at each step, good for low-entropy tasks like translation but prone to dull, repetitive text in open-ended generation.
Key takeaway: Decoding controls how much you trust the distribution's peak versus its tail: temperature reshapes it, top-k/top-p truncate it, beam search abandons sampling entirely for high-probability sequences.
| Strategy | Mechanism | Deterministic? | Best for |
|---|---|---|---|
| Greedy () | Always argmax | Yes | Evals, tests, consistency-critical tasks |
| Temperature sampling | Softmax over | No | General chat/creative generation |
| Top-k | Fixed candidate count | No | Simple truncation baseline |
| Top-p (nucleus) | Dynamic set with mass | No | Robust default across confidences |
| Beam search | Tracks best sequences | Yes | Translation, summarization |
Read the implementation
1import torch
2import torch.nn.functional as F
3
4def sample(logits: torch.Tensor, temperature=1.0, top_k=0, top_p=0.0) -> int:
5 logits = logits / max(temperature, 1e-8) # temperature scaling
6 if top_k > 0: # keep k highest logits
7 kth = torch.topk(logits, top_k).values[..., -1, None]
8 logits = logits.masked_fill(logits < kth, float("-inf"))
9 probs = F.softmax(logits, dim=-1)
10 if top_p > 0.0: # nucleus: keep cumulative >= p
11 sorted_probs, idx = torch.sort(probs, descending=True)
12 cutoff = torch.cumsum(sorted_probs, dim=-1) > top_p
13 cutoff[..., 0] = False # always keep the top token
14 sorted_probs[cutoff] = 0.0
15 probs = torch.zeros_like(probs).scatter(-1, idx, sorted_probs)
16 probs = probs / probs.sum()
17 return torch.multinomial(probs, 1).item()
18
19logits = torch.randn(50000)
20token = sample(logits, temperature=0.8, top_p=0.9)Questions and trade-offs
- Conceptual: What does temperature do to the next-token distribution? (Scales logits before softmax: low T sharpens toward greedy, high T flattens toward uniform/more random.)
- Implementation: What's the difference between top-k and top-p sampling? (Top-k keeps a fixed number of candidates; top-p keeps a dynamic number whose cumulative probability reaches p: adapting to the model's confidence.)
- Applied: When would you prefer greedy or beam search over sampling? (Low-entropy, single-correct-answer tasks like translation or summarization, where you want the most probable sequence rather than diversity.)
- Systems-level: What does temperature = 0 give you, and why is it useful? (Deterministic greedy decoding: reproducible outputs, useful for evals, tests, and tasks needing consistency.)
- Failure modes: Why can beam search produce dull or repetitive text in open-ended generation? (It optimizes for high total probability, which favors safe, generic continuations and degenerate repetition over natural diversity.)
Check your understanding
From memory: write the temperature-scaled softmax, explain top-k vs. top-p in one sentence each, and name a task where greedy/beam beats sampling. Check against Stage 3.
Oliver Perrin
Machine Learning Engineer · Founder, LiminalML
LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.
Keep exploring the idea.
Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.
Related concepts
LLMs
KV Cache
See which token projections caching saves, what attention must still read, and how memory use grows with context.
Deep Learning
Transformer Architecture
Follow attention, feed-forward layers, normalization, and residual connections through a Transformer block.
LLMs
Tokenization & BPE
Follow a pair of vocabulary merges and distinguish learning merge rules from applying a tokenizer to new text.