Verification, Not Sampling, Limits Test-Time Scaling for AI Agents

Azalia Mirhoseini frames verification as the constraint on useful test-time scaling. A language model may be able to generate a correct response to a difficult question—especially when repeated sampling gives it many chances to explore—but that is different from selecting the correct response reliably from the resulting set.

This is the generation-verification gap. More samples can raise coverage: the chance that a correct answer appears somewhere among the candidates. They do not by themselves improve selection. Majority voting, or self-consistency, offers a cheap baseline: generate many solutions, group them by final answer, and return the most common one. But a correct trajectory can be rare, and adding candidates can make a selector’s job harder once the set contains many plausible but subtly flawed answers.

Verification recasts the task. Rather than ask which answer is most frequent, a system estimates which candidate is correct. That estimate may come from an outcome reward model that scores a full solution, a process reward model that scores intermediate steps, an LLM judge, a deterministic tool, or an ensemble of such signals. The verifier can rerank candidates at inference time; it can also become a reward signal for training the generator itself.

The first operating question is whether the generator is producing candidates worth selecting among. In Cobbe et al.’s comparison of generator and verifier sizes, a large generator with a small verifier performed substantially better than a small generator with a large verifier. Mirhoseini identifies the best allocation of capacity between generation and verification as an open research question, but the result establishes a practical constraint: verification only improves the answers that generation makes available.

Cobbe et al.’s results also show why more sampling is not monotonically valuable. Solve rate improved as the number of completions per problem rose, with substantial gains through roughly 400 completions. Beyond that point, performance declined. The candidate set may still contain correct answers, but the verifier’s ranking precision deteriorates: among many close solutions, an incorrectly scored response can receive the maximum score and displace the correct one.

The paper used 100 samples in practice, where it received most of the benefit without the instability at higher sample counts. A learned verifier extended the useful range of inference-time sampling beyond majority voting, which Mirhoseini says failed at far smaller sample counts in the preceding discussion of test-time scaling. It did not eliminate the underlying selection problem.

That produces the central systems question running through these methods: given a fixed inference budget, should a system spend its compute on more candidate generations, stronger or more numerous verifiers, or deeper verification of each candidate? The source does not settle that allocation. It repeatedly shows that both sides of the pipeline matter—and that gains in one can be wasted by weakness in the other.

The 2021 OpenAI work on math word problems established two foundations for verifier training: GSM8K, a dataset of roughly 8,500 grade-school math problems, and a learned model that estimates whether a generated solution is correct.

GSM8K was designed to expose reasoning failures that do not require advanced mathematics. Its problems involve elementary operations but require a model to interpret a situation, set up the calculation, and carry it through over several steps. In the pizza example used in the source, a model calculates that six friends consume 36 cheese slices, then adds only eight pepperoni slices rather than multiplying pepperoni consumption across the six friends. It confidently concludes that Kate needs three pizzas.

The outcome-verification pipeline turns known final answers into training data. A generator is fine-tuned on the task and sampled repeatedly—100 completions per training problem in Cobbe et al.’s setup. Each completion is marked correct or incorrect by comparing its final answer with the human-provided answer. Those question-solution-label triples train a separate verifier.

At inference time, the system generates multiple candidate solutions, scores each one, and returns the candidate with the highest verifier score. This does not make the generator intrinsically more reliable. It exploits a more modest condition: the generator sometimes produces the right answer, while the verifier can identify it better than a naive selection rule can.

The verifier in Cobbe et al. is itself a language model with a small scalar prediction head. Its training objective combines correctness prediction with the usual next-token language-modeling objective. The paper compares a single score conditioned on the whole solution with token-level predictions that culminate in a score for the full completion. In the examples discussed, a correct solution remains favorably scored through its conclusion, while an incorrect spending calculation turns unfavorable near the point where the arithmetic fails.

Outcome supervision has a practical advantage when a task has a dependable terminal check. A final mathematical answer can be compared against ground truth; code can be tested; a structured output can be validated. The limitation is coarse credit assignment. A correct final answer can conceal invalid reasoning, while an incorrect answer says little about where a trajectory went wrong.

In Cobbe et al.’s comparison, verifier-based selection outperformed direct supervised fine-tuning as the verifier training set grew, for both 6B and 175B models. The benefit was conditional. With small datasets, verification was not useful and could overfit; for the 175B comparison, there was little benefit below roughly 1,000 training examples. With more data, selection by a verifier pulled ahead.

For tasks with reliable terminal checks, outcome supervision offers a direct operating path: generate candidates, label them from final-answer correctness, and test whether verifier-based selection improves the end-to-end system. It is not a full account of reasoning quality, but it creates a useful learned selector without requiring a judgment on every intermediate step.

An outcome reward model, or ORM, assigns one reward to a completed solution. A process reward model, or PRM, scores each intermediate reasoning step. The distinction matters when a correct final answer is not enough evidence that the route to it was sound.

A model can arrive at the correct answer after invalid reasoning, accidental cancellation of errors, or a lucky guess. Outcome supervision would label that completion positively. Process supervision asks whether the intermediate work is locally acceptable and can provide more precise credit assignment when a trajectory first goes wrong.

In the formulation discussed for Let’s Verify Step by Step, the final score for a process-supervised solution can combine the estimated correctness probabilities across its steps:

The study’s PRM800K dataset contains 800,000 step-level labels across 12,000 math problems.

The label-collection strategy focused on “convincing wrong answers”: solutions that the current process model rated highly but whose final answers were wrong. Rather than spend annotation effort on random samples, the researchers used active learning to surface cases where the model’s own judgment was most misleading. The study found this to be 2.6 times more data-efficient than random sample selection.

A fraction example makes the difference concrete. A solution correctly defines the numerator as $x$, states that the denominator is $3x-7$, and derives $5x=6x-14$. It then concludes $x=7$, though the correct answer is 14. Outcome supervision records only that the full solution failed. Step-level labels preserve that the setup was reasonable and the final algebraic conclusion was not.

The study fine-tuned both ORM and PRM models from GPT-4. ORM data paired completions with final-answer correctness; PRM data paired completions with PRM800K step labels. Process supervision outperformed outcome supervision and majority voting as the number of candidate solutions increased. The gap widened with more samples, and the PRM could identify correct solutions on some problems where fewer than 5% of generated candidates were correct.

The reported out-of-distribution results also favored PRM over ORM and majority voting across the compared subjects.

Process supervision is not a universal replacement for outcome supervision. A student raised the possibility that locally plausible steps can still fail to produce a valid final answer. Mirhoseini’s answer was that newer systems often combine PRM and ORM signals, seeking the benefits of both. A PRM also introduces operational choices—such as score thresholds—that must be tuned.

There is a further distinction between using a PRM to select outputs and using it to train the generator. A PRM can inspect the question, prior reasoning, and current step without changing the generator. The generator can be prompted to provide steps, then the verifier can rank the result. Mirhoseini noted a possible failure mode once a PRM is used to fine-tune a generator: the model could learn to emit answers or patterns the PRM favors rather than preserve the intended reasoning process.

Math-Shepherd addresses the main cost of PRMs: obtaining human judgments for intermediate steps. Its automatic annotation method defines the quality of a reasoning step by its potential to reach a correct final answer.

For a given intermediate step, the system samples multiple continuations and checks their final answers against ground truth. It can assign a hard estimate, which marks a step successful if any continuation reaches the right answer, or a soft estimate, which records the share of continuations that finish correctly. If three continuations begin after a step and two reach the correct answer, the soft estimate is $\frac{2}{3}$; the hard estimate is $1$.

This is not direct process supervision. It replaces a human statement—“this step is good”—with a computational proxy: “some sampled continuation from this point can still succeed.” The distinction is operationally important. Rollout labels fit tasks where final answers are cheap to check and the system can afford to sample trajectories. They are not a general substitute for human judgments about whether a step itself is correct.

A promising but unusual path can be assigned a low score if the rollout budget never discovers its successful continuation. Hard problems may yield no positive signal when none of the sampled continuations reaches the answer. And a flawed step can receive a favorable label if a later trajectory nevertheless arrives at the correct result. Mirhoseini made both problems explicit in discussion: a small rollout budget can miss a valid path, and wrong intermediate work can still lead to a correct final answer.

More rollouts can reduce these errors without guaranteeing that a label reflects actual step quality. Math-Shepherd found that larger numbers of decoded paths improved training loss, while its best verifier performance arrived at four paths. It found no substantial downstream difference between hard and soft annotations and ultimately used the simpler hard estimate.

The resulting auto-annotated PRM was evaluated as a reranker, as a reward model for PPO-based reinforcement learning, and in combination. It exceeded self-consistency and ORM on the reported GSM8K and MATH comparisons; on MATH, it also exceeded PRM800K, the human-annotated process model used as a baseline.

The training loop compounds the role of the verifier. A generator produces trajectories; sampled continuations create automatic step labels; those labels train a PRM; and the PRM then supplies rewards for PPO to optimize the generator. In the reported greedy-decoding results, Math-Shepherd’s step-by-step PPO exceeded both RFT and ORM-guided PPO. For LLaMA2-7B, the source reports GSM8K and MATH scores of 73.2 and 21.6 for step-by-step PPO, compared with 70.8 and 20.8 for ORM-PPO. For Mistral-7B, the corresponding scores were 84.1 and 33.0, compared with 81.8 and 31.3.

Adding verification on top of PPO improved the reported Mistral-7B results further, though Mirhoseini noted that the optimization gains appeared to plateau. The unresolved systems question remains the same: under a fixed compute budget, should a model generate more candidates, run more rollouts from intermediate steps, or spend more effort judging the candidates already produced?

Weaver takes a different route from training a single new reward model. It combines multiple imperfect existing verifiers: PRMs, ORMs, and language-model judges prompted to assess a candidate response.

A weak verifier is not deliberately poor. It is any function that correlates with correctness but remains imperfect. The premise is that multiple weak signals can produce a stronger selector if they contribute distinct information. If every verifier agrees on every example, adding models provides little value.

Weaver uses a score, weight, select procedure. It gathers outputs from several verifiers, normalizes them to a common scale, and filters out low-quality models. It then estimates weights for the remaining verifiers using weak supervision and limited labeled data—about 1% in the low-label configuration discussed—before combining their scores to select the most confident answer.

The approach assumes verifiers capture independent aspects of correctness. Under that assumption, the posterior probability of correctness can be represented as:

The independence assumption is not a claim that models are truly unrelated. It is the modeling condition that lets agreement and disagreement among noisy verifiers inform estimates of their reliability. Weaver uses simple weighting approaches, including Naive Bayes and logistic regression, to combine those signals.

Across MATH500, GPQA Diamond, MMLU, and MMLU Pro, weighted ensembles outperformed naive averaging in the compared configurations. Weaver also outperformed majority voting and a multi-agent verification baseline as repeated generation scaled. That multi-agent approach used off-the-shelf models as aspect-specific binary judges without training; it did not match the weighted ensemble results on the tasks shown.

An 8B Llama 3.1 generator paired with verifiers of 8B and below reached a 70.0% average, close to the 72.2% shown for majority voting with a 70B Llama 3.3 generator. Applying Weaver to the 70B generator reached an 86.2% average, close to the reported 86.7% first-sample result for o3-mini. These figures concern final solution accuracy for the whole generate-and-select system, not merely whether a correct answer appeared in the candidate set.

The ensemble’s limitation is inference cost. Every candidate may need to be scored by many reward models and LLM judges, a cost that grows with the number of generated candidates. Weaver’s proposed answer is distillation: train the full ensemble as a teacher, then train a smaller verifier to imitate its selection behavior.

The reported distilled model had 400M parameters, compared with a 70B-scale verifier ensemble. It preserved more than 97% of the reported accuracy gains while using 99.97% less verification compute.

Distillation addresses the cost of an ensemble after its selection quality has been established. More judges alone are not the method: Weaver depends on filtering, weighting, and verifiers whose signals are not redundant.

The methods discussed do more than rerank answers. A trained verifier can become the reward signal in a reinforcement-learning loop: generate solutions, identify positive trajectories, use them to improve the generator, and repeat. Mirhoseini says reasoning models already use versions of this pattern, generating reasoning traces and using reward-guided reinforcement learning to produce data for subsequent rounds.

That does not make test-time exploration obsolete. A model can still benefit from sampling because repeated attempts explore different regions of the solution space before a selector chooses among them. But Mirhoseini describes stronger Pass@1 performance as a desirable direction: a model that produces a good answer on the first attempt, reducing the need for costly search and verification at deployment.

She also raises a possible tension in pushing too far toward one sharply preferred answer. Making a model’s probabilities highly concentrated around a single answer could, she suggests, reduce the creativity and diversity of solutions that make exploration valuable. The tension is not resolved in the source, but it shapes the goal of distilling search and verification into training-time capability.

The verification framework is not limited to grade-school mathematics. Mirhoseini identifies coding as a useful domain and points to an alternative design: have a model generate unit tests, then use those tests as verifiers rather than train a conventional reward model. Tool use, task-specific rubrics, calculators, and symbolic systems such as SymPy can similarly add checking signal for errors that are mechanically testable.