Train-Time Scaling Makes Verifiable Reasoning More Consistent, Not Necessarily More Capable

? aakanksha-chowdhery defines train-time scaling as a feedback loop: generate model outputs at inference time, filter or score them, then use successful outputs to train the model that generated them. In the verifiable reasoning settings under discussion, the aim is to make correct reasoning more likely to appear directly, rather than relying entirely on repeated search at deployment.

The central constraint is more specific than “apply RL to reasoning.” Train-time scaling needs a base model capable of producing or reconstructing some useful solutions, plus a training setup that yields meaningful reward variation among those solutions. Without initial competence, there is little credible material to bootstrap. Without variation, there is no useful direction for a relative learning method to reinforce.

On AIME 2024, Chowdhery cites roughly 5% accuracy for GPT-3.5, 51.7% for DeepSeekMath-RL 7B, and 50% for DAPO applied to Qwen-32B. The comparison is meant to show that a smaller model can become highly effective on a verifiable reasoning benchmark through training compute and feedback loops, not only through more parameters.

The relevant domains are those where a system can distinguish good outputs from bad ones. In mathematics, the final answer can often be checked against a known solution. In programming, a system can run code, use execution feedback, or evaluate unit tests. Chowdhery’s domain-comparison slide also places data analysis among the areas where reasoning models show larger gains than they do in personal writing or text editing.

Test-time scaling and train-time scaling use this feedback differently. At test time, a model can repeatedly sample answers and use a verifier to choose among them. Chowdhery characterizes this as throwing “spaghetti at the wall” when the system has a reliable way to identify where it should land. Train-time scaling uses successful trajectories to make correct reasoning more likely before that search begins.

Both training compute and inference compute improved pass@1 on the o1 AIME scaling curvesThe displayed o1 curves rise as either train-time or test-time compute increases on AIME. shown in the lecture. Chowdhery does not present those curves as a universal rule for allocating compute. Test-time search is especially powerful when verification is robust; train-time scaling matters when the objective is to increase the chance that the model emits a useful answer before a large search is required.

Reasoning models allocate tokens to intermediate work instead of immediately producing an answer. Chowdhery identifies recurring behaviors in those traces: interpreting the problem, decomposing it into subproblems, checking intermediate work, correcting an error, and trying an alternative approach.

A displayed coding example asks for a Bash script that transposes a matrix represented as [1,2],[3,4],[5,6]. The model first spells out the input and desired output format, an instance of problem analysis. It then proposes a sequence of steps: parse the string, construct a matrix, transpose it, and format the result. The decomposition is unnecessary for a small example but provides a pattern for handling harder tasks.

A chemistry example shows a model calculating the pH of an ammonium fluoride solution. Its reasoning begins with one formula, pauses at “Wait, the correct formula is,” and switches to an expression involving the relevant acid and base constants. Chowdhery uses such behavior to illustrate self-correction within a visible chain of thought: the model can generate an alternative calculation before it commits to a final answer.

The value of these traces is highest where they can be checked. Chowdhery’s chart comparing o1-preview with GPT-4o shows larger human-preference gains in computer programming, data analysis, and mathematical calculation than in personal writing or editing. Her explanation is that the first group offers stronger ways to verify outputs and close a feedback loop.

That does not mean every longer trace is useful. A model can produce repetitive or broken chains, which Chowdhery calls overthinking. Nor does a correct final answer guarantee that every intermediate step was sound. Train-time scaling can make a model more coherent and more reliable at emitting paths already available to it; whether it expands the set of problems the model can solve is a separate question.

STaR, or Self-Taught Reasoner, begins with a data shortage. Large-scale internet corpora contain much more text and many more answers than explicit step-by-step rationales. Human-written rationales are expensive. Template-generated reasoning applies only where solution patterns are already known, and a small set of few-shot rationale examples can underperform direct fine-tuning on a much larger dataset.

STaR uses a small prompt set of question-rationale-answer triples to generate a larger training set. The broader dataset need only contain questions and correct answers.

The model is prompted with the seed rationales and asked to solve each training question. When its answer matches the known answer, STaR retains the generated rationale and answer as a training triple. The model is then fine-tuned on the collected successful traces and the process repeats.

The distinctive step is rationalization. When the model answers incorrectly, STaR supplies the correct answer as a hint and asks the model to explain how one gets there. Chowdhery summarizes the move as: “The answer is 42. Now show your work.” If the model produces a rationale that leads to the known answer, that rationale enters the next training set—but the hint is removed during fine-tuning, so the model learns from the example as if it had solved it directly.

Rationalization addresses the problem with a direct-success-only loop: it would learn mostly from questions the model can already solve. Giving the answer lets the model search for explanations conditioned on both the question and the destination. Fine-tuning on those traces can make the next round of unaided solutions stronger.

The mechanism rests on three important assumptions.

First, STaR treats final-answer correctness as a proxy for rationale quality. Chowdhery says this was generally useful for math, but it can retain a flawed chain that reaches the right answer and discard a partially valuable chain that reaches the wrong one.

Second, the model must be able to generate a valid backward rationale once it receives the answer. STaR itself does not add a second filter over rationalized chains. Chowdhery notes that later work can use process reward modelsA model that scores intermediate reasoning steps rather than only a final answer., or humans can inspect chains. Those safeguards are outside the original method.

Third, the initial model must be capable enough to bootstrap. If a problem class is completely outside its range, it may neither solve examples directly nor produce a valid rationale when given a hint. Iteration can build outward from partial competence; it does not guarantee a logical leap into an arbitrary new domain.

The original experiments used GPT-J, a 6B-parameter open model, on GSM8K, CommonsenseQA, and synthetic multi-digit arithmetic. The training schedule used a short warmup, a constant learning rate, and progressively more inner-loop steps across outer iterations. Chowdhery highlights the paper’s observation that a slower start to training benefited performance.

The strongest rationalization result in the presentation is CommonsenseQA. GPT-J directly fine-tuned on all training data reached 60.0% development-set accuracy. STaR without rationalization reached 68.8% while using 69.7% of the data; STaR with rationalization reached 72.5% while using 86.7%. The authors also reported that human raters were 30% more likely to rank STaR-generated rationales above the initial few-shot rationales.

GSM8K produced a different pattern. Few-shot GPT-J reached about 3.0%, direct fine-tuning 5.8%, STaR without rationalization 10.1%, and STaR with rationalization 10.7%. STaR improved over the baselines, but rationalization added little beyond direct self-generated reasoning. Chowdhery’s interpretation is that this grade-school arithmetic task was sufficiently within the model’s range that backward-generated rationales did not add much new structure.

STaR also plateaus. Chowdhery cites a range of roughly 16 to 36 iterations and describes the method as sophisticated filtering and fine-tuning rather than full online RL. It resets to the base model at each iteration, which the presentation identifies as wasteful.

The method remains useful because it can construct a rationale dataset without requiring people to write every chain of reasoning. But it inherits the style and biases of its few-shot prompts, and rationalization can amplify invalid or biased explanations that happen to end at the correct answer. A student noted that a stronger frontier model could instead provide traces through distillation; Chowdhery called that a fair point, while describing STaR as an attempt to build self-improvement from a model’s own outputs.

DeepSeekMath improved the base model before applying RL. Chowdhery says its authors challenged the conventional focus on arXiv papers as the primary mathematical data source. They began with OpenWebMath’s 13.6 billion tokens, trained a fastText classifier to identify math-like text, mined Common Crawl, and found 120 billion tokens. They then iterated by identifying more mathematical domains and retraining the classifier.

The point was coverage and curation rather than source prestige. Chowdhery says arXiv-focused training could sometimes hurt performance, while beginning from a code-oriented model and moving from code to math helped tool use and reasoning. The model entered RL with a stronger mathematical base.

DeepSeekMath’s main RL contribution was Group Relative Policy Optimization, or GRPO. PPO-style RL can require an original policy, the policy currently being trained, a critic or value model, and a reward model. Holding these components becomes expensive as models scale.

GRPO removes the critic. For a given question, the system samples a group of answers—64 in Chowdhery’s simplified example—scores them, and uses the group’s reward distribution as the baseline. A completion’s advantage is its reward minus the group’s average reward, divided by the spread of rewards in that group.

A completion that scores better than its peers receives positive advantage; one that scores worse receives negative advantage. Since reward models are often trained from comparisons, the group supplies a natural comparative context. Removing the critic also saves the memory required to hold and train it.

Chowdhery presents the distinction among related approaches in terms of how they assign a gradient coefficient. STaR generates a response and applies a binary outcome: one for correct and zero for wrong. Online rejection fine-tuning samples from the current model but still retains only binary correctness. GRPO samples online and applies a continuous, group-relative advantage.

The method needs a group with mixed outcomes. If all samples are correct or all are wrong, the normalized advantage provides no useful contrast. This is why examples near a model’s current boundary are useful for this type of learning: they produce differences among sampled answers that the method can reinforce.

On the MATH benchmark, Chowdhery cites an increase from 46.8% to 51.7%, describing DeepSeekMath as the first open-source 7B-scale model to cross 50%. The important behavioral result came from the online RL loop: sampling from the current model outperformed sampling from an older policy.

But the paper’s reported improvement was in majority@K rather than pass@K. Across 32 samples, the majority answer became more likely to be correct. The chance that at least one of the 32 samples was correct did not improve in the result Chowdhery presents. Her interpretation is that the model became more consistent rather than demonstrably more capable in a fundamental sense: training made solutions already reachable in its samples more likely to appear and win.

DAPO addresses the instability that appears when GRPO is pushed to longer reasoning chains and harder mathematical problems. Chowdhery reports that naive GRPO on Qwen-32B reached 30% on AIME, while DeepSeek-R1-Zero-Qwen-32B reached 47%. DAPO makes the operational fixes explicit: preserve exploration, prevent response length from running away, and maintain batches with usable reward variation.

The first change is Clip-Higher, an asymmetric version of PPO-style clipping. Standard clipping treats probability increases and decreases symmetrically. Chowdhery’s account is that this can prevent low-probability tokens from increasing enough to support exploration while familiar, high-probability tokens continue to dominate. The model becomes overly confident and entropy collapses.

Clip-Higher allows larger probability increases than decreases. In the presented progression, it moved AIME performance from 36% to 38% after overlong filtering had already been applied. The accompanying entropy curve remained more stable. Chowdhery treats entropy as a practical proxy for exploratory capacity: too low means collapse; too high means incoherent output.

Dynamic sampling directly addresses a practical limit of group-relative learning. An all-correct group or all-wrong group has no relative advantage signal. Chowdhery says that as training proceeds, 60% of questions may eventually produce 64 correct samples. Continuing to train on such groups spends compute without producing a useful gradient.

DAPO oversamples, then retains groups where the correct count is strictly between zero and the group maximum. The objective is to keep an effective batch size of examples that can actually contribute a learning signal.

The remaining interventions manage length. With sample-level loss, a long, poor completion can receive the same aggregate training weight as a short, useful one. DAPO uses token-level loss instead, which Chowdhery says better controls response length and improved the reported score from 41% to 42%.

The lecture’s entropy and response-length charts support the operational concern: without token-level loss, generated responses and entropy grew without control; with it, both were more stable. Soft overlong punishment handles a related failure mode. A hard cutoff at the maximum context length can turn an otherwise promising but long chain into a noisy training example. DAPO applies a gradual penalty in the final 4,000-token interval before the limit. Other approaches may increase context length over the RL loop; DAPO instead shapes the penalty.

The progression from 30 to 50 shows why Chowdhery stresses implementation details. The base model and broad RL objective remain similar, but outcomes change materially when exploration, response length, truncation, and reward distribution are controlled. The final DAPO result of 50 exceeded the 47 cited for DeepSeek-R1-Zero-Qwen-32B.

Chowdhery’s practical guidance is to monitor more than training loss. Response length should grow without exploding. Entropy should remain between collapse and noise. The fraction of fully correct samples indicates when sampling needs adjustment. If improvement stops after roughly 2,000 steps, the reward model may have saturated.

This is also the tradeoff between supervised fine-tuning and RL. RL can reinforce behavior from relatively few examples when rewards are strong and verifiable, but the optimization loop is fragile. Supervised fine-tuning can be the faster route when abundant high-quality data exists. RL becomes attractive when the system can use reward signals to reinforce behavior that supervised examples alone do not reliably elicit.

STaR, GRPO, and DAPO correspond to different practical constraints. Chowdhery presents STaR as useful with fewer than roughly 100 rationale examples, limited compute, no RL infrastructure, and relatively simple reasoning tasks. GRPO fits a capable base model with good instruction data, standard mathematical reasoning tasks, and memory constraints that make a critic expensive. DAPO is aimed at long reasoning chains, competition-level problems such as AIME and IMO, and settings that can support careful RL monitoring.

The operating test is whether the loop has a foothold: a model that can already solve or rationalize some relevant cases, a verifier or reward model that can distinguish useful outputs from bad ones, and enough variation in sampled outcomes to produce a learning signal. Chowdhery frames weak or hackable rewards as a central obstacle: if a reward model lacks sufficient signal, the loop cannot reliably improve; if it can be gamed, optimization can move in the wrong direction.

That framing makes majority@K versus pass@K the lecture’s central open problem. The techniques discussed improve majority@K, answer formatting, and multi-step coherence. But Chowdhery says they do not yet reliably improve pass@K, novel problem-solving, or out-of-distribution generalization. If a model already emits a correct answer somewhere in a set of samples, training may make that answer more consistent or more likely to win. It does not necessarily show that the model has acquired a solution to problems it could not previously reach.

The remaining questions follow directly. Are visible behaviors such as backtracking and self-correction learned procedures, or patterns already latent in the base model that become more frequent? How can a system learn from failures rather than simply filtering them out? Can STaR’s rationalization be combined with DAPO’s stability techniques?

Verification remains the limiting resource. Math and code are favorable because they can provide final-answer checks, execution feedback, and unit tests. Chowdhery also points to ensembles of verifiers as a possible way to compensate for gaps in a single signal. The relevant question is not simply whether there is a large dataset, but whether there are enough relevant examples and reliable signals to improve without filtering away the cases that create a useful gradient.