Verifiers for Reasoning Models

A verifier decides whether a model's answer is correct without asking a human. For math, it extracts the final answer, normalizes it, and checks equivalence with a reference. For code, it runs tests. The same function serves two jobs: evaluation (what fraction of a benchmark does the model solve?) and reward (the signal RLVR training optimizes).

That double role is the reason to build it carefully. A verifier measures only what it checks. A lenient parser inflates benchmark scores; during RL, the policy will find and exploit the same leniency.

A reasoning model writes a long solution and ends with something like \boxed{\frac{3}{4}}. The reference answer is 0.75. A string comparison says wrong. A useful verifier finds the final boxed expression, strips formatting, converts both sides to a canonical form, and only then compares.

Every step can fail in both directions. Extraction can grab an intermediate number. Normalization can merge answers that are not equal. Equivalence checking can time out on a hard symbolic expression. Log each failure type separately so that a change in score can be traced to the model or to the checker.

Treat the verifier as a classifier $v(x, y) \in \{0, 1\}$ of whether response $y$ answers prompt $x$ correctly. Against true correctness $c(x,y)$, it has a false-negative rate $\alpha = P(v=0 \mid c=1)$ and a false-positive rate $\phi = P(v=1 \mid c=0)$. If the model's true accuracy is $a$, the measured accuracy is

so with no false negatives, a 5% false-positive rate on a model with 40% true accuracy adds 3 points of measured accuracy that do not exist. In RL, the expected reward of a policy is exactly this $\hat a$, so the gradient also rewards whatever raises $\phi$.

DeepSeek-R1-Zero used two rule-based rewards: an accuracy reward from checking the final answer, and a format reward for placing reasoning between thinking tags. The paper reports avoiding neural reward models for this stage because of reward hacking at scale.

Key takeaway: Report the verifier with the score. A benchmark number is a property of the model and the checker.

Outcome versus process checks

An outcome verifier scores only the final answer, so a correct answer reached by flawed reasoning still earns full credit. A process reward model scores intermediate steps, which gives denser feedback but is a learned model with its own errors. Lightman et al. compared the two on MATH; the MATH-500 split commonly used for evaluation comes from that work.

Extract the last \boxed{...} with brace matching, then compare numerically where possible.

1from fractions import Fraction
2
3def last_boxed(text: str) -> str | None:
4    start = text.rfind("\\boxed{")
5    if start < 0:
6        return None
7    i, depth = start + len("\\boxed{"), 1
8    for j in range(i, len(text)):
9        depth += {"{": 1, "}": -1}.get(text[j], 0)
10        if depth == 0:
11            return text[i:j]
12    return None                                  # unbalanced braces: no answer
13
14def canonical(ans: str):
15    s = ans.replace(" ", "").replace("\\left", "").replace("\\right", "").strip("$")
16    if s.startswith("\\frac{") and "}{" in s and s.endswith("}"):
17        num, den = s[len("\\frac{"):-1].split("}{", 1)
18        s = f"{num}/{den}"
19    try:
20        return Fraction(s)
21    except (ValueError, ZeroDivisionError):
22        return s
23
24def verify(response: str, reference: str) -> int:
25    pred = last_boxed(response)
26    return int(pred is not None and canonical(pred) == canonical(reference))
27
28print(verify("... so the answer is \\boxed{\\frac{3}{4}}", "0.75"))  # 1

This handles a narrow set of forms. Production math verifiers add symbolic equivalence with a timeout, unit handling, and an explicit list of known false positives as regression tests.

1. Conceptual: Why use the same verifier for evaluation and RL rewards? (It keeps the training signal and the measurement consistent, but any verifier error then shapes both the model and its reported score.)
2. Applied: Measured accuracy jumped after a verifier change. What do you check? (Rescore a fixed set of saved responses with both versions and inspect every flipped label.)
3. Implementation: Why take the last boxed answer? (Reasoning traces often box intermediate results; the final one is the committed answer. This is a convention, so state it.)
4. Systems-level: What must a code verifier isolate? (Execution: time limits, memory limits, no network or file system access beyond the sandbox.)
5. Failure modes: How can a format reward be hacked? (The policy can emit the required tags around empty or irrelevant reasoning. Keep format rewards small relative to accuracy.)

From memory: write measured accuracy in terms of true accuracy and the two verifier error rates, and list the three verifier steps. Check against Stage 3.

Sources and further reading

• Lightman et al., Let's Verify Step by Step: outcome versus process supervision, and the source of the MATH-500 split.
• Hendrycks et al., Measuring Mathematical Problem Solving with the MATH Dataset: the MATH benchmark.
• DeepSeek-AI, DeepSeek-R1: rule-based accuracy and format rewards.
• Raschka, Building Reasoning Models from the Ground Up: builds a math verifier, then reuses it as the RLVR reward.