Scoring Answers with Log-Probabilities

A language model assigns a probability to every token it generates. Adding the log-probabilities of a response's tokens gives the log-probability of the whole response under the model. That single number can rank candidate answers, flag low-confidence outputs, and act as a scoring function for best-of-N sampling when no verifier exists.

It measures how expected the text is to the model, not whether it is correct. A fluent wrong answer can score higher than an awkward right one, and the way you aggregate tokens decides which kinds of answer win.

Each token's surprisal is $-\log p$: near zero for a token the model expected, large for one it did not. Shading text by surprisal shows where the model hesitated, which is often the number or claim that matters. Summing surprisal over a response gives its total cost; dividing by its length gives an average.

Switch between the two scoring rules below. Answer A is correct but uses five tokens. Answer B is wrong but uses one.

For a prompt $x$ and response $y = (y_1,\dots,y_T)$, the sequence log-probability factorizes over tokens:

Every term is at most zero, so adding tokens can only lower the total: sum scoring prefers short responses. The length-normalized score divides by $T$:

In the example, answer A totals $-1.688$ and B totals $-1.204$, so the sum selects the wrong, shorter answer. Per token, A averages $-0.338$ and B $-1.204$, and the mean selects A. Intermediate choices, such as dividing by $T^\alpha$ with $0<\alpha<1$, trade the two biases against each other.

Without an end-of-sequence token, the sum measures the probability of that token prefix, not the probability that the model stops there. Include EOS when scoring a complete stopped response, and keep the tokenizer, prompt, answer span, and stopping convention consistent across candidates. The mean is a ranking heuristic, not a sequence log-probability.

Key takeaway: Log-probability is a confidence signal from the model about its own text. Use it to rank or route, and pair it with a verifier when correctness matters.

Score the answer, not the whole response

For math problems, the reasoning before the answer can be long and varied, while the final answer is a few tokens. Scoring only the answer span asks how confident the model is in its conclusion, given its own reasoning. That is closer to what you want for selection, but it still rewards a confident mistake.

Score a response with one forward pass over prompt plus response. Logits at position $t$ predict token $t+1$, so shift by one and keep only the response positions. Supply a nonempty prompt (or a beginning-of-sequence token) and a nonempty response; include EOS in the response ids if you want to score stopping.

1import torch
2import torch.nn.functional as F
3
4@torch.no_grad()
5def response_logprob(model, prompt_ids, response_ids, normalize=True):
6    ids = torch.cat([prompt_ids, response_ids])[None]          # [1, P + T]
7    logits = model(ids).logits[0, :-1]                          # predict next tokens
8    logp = F.log_softmax(logits.float(), dim=-1)                # float32 for stable log-softmax
9    targets = ids[0, 1:]
10    token_logp = logp.gather(1, targets[:, None]).squeeze(1)
11    response = token_logp[prompt_ids.numel() - 1:]              # response tokens only
12    return response.mean() if normalize else response.sum()

Compute log-softmax in at least float32. Very confident tokens have log-probabilities extremely close to zero, which low precision rounds to exactly zero.

1. Conceptual: Why does summed log-probability favor short answers? (Each token adds a term that is at most zero, so every extra token can only lower the total.)
2. Applied: When would you score only the answer span? (When the reasoning before it varies in length and style, and the selection question is how confident the model is in the conclusion.)
3. Implementation: Why shift logits by one position? (The logits at position t are the model's prediction for token t+1.)
4. Systems-level: Why can't you compare perplexity across models with different tokenizers? (Perplexity is per token; a different tokenizer splits the same text into a different number of tokens.)
5. Failure modes: What does a high log-probability fail to tell you? (Whether the answer is correct. It measures how expected the text is to the model, which can include its systematic errors.)

From memory: write the sequence log-probability as a sum over tokens, explain the length bias of the sum, and define perplexity from the mean. Check against Stage 3.

Sources and further reading

• Raschka, Building Reasoning Models from the Ground Up: scoring candidates with token and sequence log-probabilities, then self-refinement.
• Wu et al., Google's Neural Machine Translation System: length normalization in beam search scoring.
• Kadavath et al., Language Models (Mostly) Know What They Know: when model probabilities are calibrated, and when they are not.