Inference-Time Scaling

Inference-time scaling spends more computation per question while the model weights stay fixed. The model can write a longer chain of thought, draw several samples, score candidates, or critique and revise its own answer. Each method changes the decision procedure around the model, not what the model has learned.

Keep three questions separate: how candidates are generated, how one is selected, and what the extra compute costs. A method that helps on math with checkable answers can do nothing, or harm accuracy, on a task where the selection rule cannot tell good answers from bad ones.

Sample the same math problem eight times at a temperature above zero. The reasoning paths differ, and some reach the right final answer. Self-consistency extracts each final answer and returns the most frequent one. Best-of-N instead asks a scoring function to choose. Self-refinement runs a loop: answer, critique, revise.

Voting only helps when the correct answer is more common than any single wrong answer. If the model is usually wrong in the same way, more samples make the vote more confidently wrong. The chart below makes that exact.

Suppose each independent sample is correct with probability $p$ and, in a simplified two-answer case, otherwise lands on one competing wrong answer. With an odd number of samples $N$, majority voting is correct when more than half the samples are correct:

For $p > 0.5$ this approaches 1 as $N$ grows. For $p < 0.5$ it approaches 0. An oracle verifier that recognises any correct sample instead gives

which rises for any $p > 0$. The gap between the two curves is the value of a better selection rule. Real verifiers sit between them: they miss some correct answers and accept some wrong ones.

Key takeaway: Sampling creates options; selection decides whether those options become accuracy. Report accuracy against tokens or samples spent, not accuracy alone.

Many wrong answers change the vote

With free-form math answers, wrong samples often scatter across different values. Plurality voting can then succeed even when $p < 0.5$, because no single wrong answer outnumbers the correct one. The two-answer formula is the pessimistic case. That is why answer extraction and normalization matter: if 1/2, 0.5 and \frac{1}{2} count as three different answers, the correct vote is split.

Majority voting needs a normalizer that maps equivalent answers to the same key. This version handles simple numeric forms only.

1from collections import Counter
2from fractions import Fraction
3
4def normalize(answer: str) -> str:
5    text = answer.strip().strip("$").replace("\\frac{", "").replace("}{", "/").rstrip("}")
6    try:
7        return str(Fraction(text).limit_denominator())  # "0.5", "1/2" -> "1/2"
8    except (ValueError, ZeroDivisionError):
9        return text.lower()
10
11def self_consistency(final_answers: list[str]) -> tuple[str, float]:
12    if not final_answers:
13        raise ValueError("Need at least one sampled answer")
14    counts = Counter(normalize(a) for a in final_answers)
15    answer, votes = counts.most_common(1)[0]
16    return answer, votes / len(final_answers)   # agreement, not calibrated confidence
17
18answers = ["1/2", "0.5", "\\frac{1}{2}", "2/3", "0.5"]
19print(self_consistency(answers))  # ('1/2', 0.8)

The agreement fraction is useful for routing (for example, sample more when agreement is low), but it is not a probability that the answer is correct.

1. Conceptual: Does inference-time scaling change model weights? (No. It changes how outputs are generated and selected at fixed weights. RL and distillation change the weights.)
2. Applied: When does majority voting make accuracy worse as N grows? (When a single wrong answer is more likely than the correct one, as in the two-answer case with p < 0.5.)
3. Implementation: Why is answer normalization part of the algorithm, not a detail? (Equivalent answers written differently split the vote and change the selected answer.)
4. Systems-level: How do you compare a method that uses 16 samples with one that uses 1? (Plot accuracy against tokens or samples spent, and include latency. Snell et al. show the best strategy can depend on problem difficulty.)
5. Failure modes: What can go wrong with best-of-N against a learned reward model? (Selection can exploit reward-model errors, choosing answers that score highly without being correct.)

From memory: write pass@N for an oracle verifier, explain why majority voting can approach zero accuracy, and name one task where self-consistency cannot be applied directly. Check against Stage 3.

Sources and further reading

• Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models: sampling diverse reasoning paths and voting over final answers.
• Snell et al., Scaling LLM Test-Time Compute Optimally: compute-optimal allocation between search and revision by problem difficulty.
• Madaan et al., Self-Refine: iterative feedback and refinement with a single model.
• Raschka, Building Reasoning Models from the Ground Up: a code-first course covering sampling, self-consistency, scoring and self-refinement on MATH-500.