Verification Is the Bottleneck for Self-Improving AI Agents

For years, the central recipe for improving large language models was straightforward: increase training compute, dataset size, and parameter count, and test loss fell according to a power-law relationship. ? aakanksha-chowdhery describes that pattern as the foundation for the progression from models such as BERT and GPT-2 through GPT-3, PaLM, Gemini, and later systems.

The lesson was not merely that bigger models performed better on familiar benchmarks. Scaling made qualitatively different uses of the same model possible. A model could move from needing task-specific fine-tuning to performing zero-shot and few-shot tasks from a natural-language prompt.

In zero-shot translation, for example, the model receives only an instruction—translate English to French—and is asked to complete a translation. Few-shot prompting adds a handful of demonstrations, but neither requires weight updates. This made prototyping much easier: rather than build and train a separate system for each task, a user could specify a task and examples in context.

Scaling also revealed capabilities that had not been reliably apparent in smaller models. Azalia Mirhoseini points to chain-of-thought prompting as the consequential case. A standard prompt may show an input and final answer; a chain-of-thought example also exposes intermediate reasoning. For a simple word problem, that means showing not just that two cans of three tennis balls added to five yields 11, but the arithmetic steps that produce the result. Larger models could use this supplied process to solve new problems more effectively.

The point is not that a model needs a long explanation to solve elementary arithmetic. Mirhoseini notes that even relatively small contemporary models can do that. The significant finding is that the underlying capability held as tasks became harder and became central to later thinking and reasoning models. Results reported for LaMDA, GPT, and PaLM found that chain-of-thought prompting offered little benefit at smaller scales, then improved math solve rates as models grew. Other tasks—including modular arithmetic, word unscrambling, Persian question answering, and grounded mappings—showed abrupt-looking gains at particular scales.

The instructors are careful about what “emergence” means. Chowdhery says chain of thought was not intentionally baked into early models by design: researchers gave larger models difficult problems and found that explaining intermediate steps improved performance. But reasoning systems today are also trained to reason more effectively. Mirhoseini’s distinction is that chain of thought was initially discovered as a capability elicited from larger models, while reasoning models are subsequently optimized to use thinking more deliberately. She frames the ability to use more reasoning when a problem needs it as a direction models are being trained toward, not as a settled property of every current system.

Pre-training teaches a model to predict the next token over broad text and data. It can give the model statistical knowledge of patterns in the world, but not necessarily an ability to follow instructions or behave in line with human preferences. Azalia Mirhoseini presents post-training as the set of steps that turn this broad predictive capability into something closer to an assistant.

The first stage can still use next-token prediction, but on more deliberately selected material: specialized math, code, creative writing, or curated instruction-following data. The objective resembles pre-training; the difference is the data. Rather than broadly absorbing text, the model is exposed to examples intended to make it more capable in particular domains and more responsive to requests.

Instruction tuning changes the training signal more directly. It supplies pairs of instructions and desired answers across many tasks. Those examples can be straightforward question-and-answer pairs, or they can include demonstrations of intermediate reasoning. The model is being supervised toward a response to a specified request: an instruction, an answer, and sometimes a process for reaching it. Mirhoseini says that the quality and breadth of this dataset materially affect the quality of the resulting model; multi-task instruction tuning is intended to generalize to tasks not included in training.

Preference optimization addresses a different problem. A supervised instruction example supplies a target response; a preference dataset asks which of several model outputs people judge better. Humans can compare answers for correctness, helpfulness, specificity, or harmlessness. Those rankings can train a reward model, which stands in for human judgments during further optimization. Reinforcement learning from human feedback, or RLHFA post-training method that uses human preferences to construct a reward signal., uses that learned reward signal to steer the language model toward generations the reward model rates highly. Mirhoseini also identifies Direct Preference Optimization as another preference-based approach.

This explains why ChatGPT represented a material shift beyond earlier scaled models. Instruction tuning, dialogue formatting, and preference training made a model more usable as an interactive system. ChatGPT reached one million users in five days, according to the launch material cited in the lecture. The technical point is that capability alone was not the whole product: the model also had to be trained to behave in ways people could direct and assess.

Alignment remains unresolved. Fine-tuning can make models more sensible and safer according to curated data and evaluation criteria, but Mirhoseini describes steering models toward human goals, values, and preferences as one of AI’s major continuing challenges—not as a solved consequence of RLHF.

Inference is the newer scaling frontier: improving the answer a fixed model produces without changing its parameters. The Large Language Monkeys work discussed in the lecture supplies the core intuition. Instead of asking a model for one solution, generate many candidate solutions and use a verifier or selection mechanism to identify a correct one.

That approach rests on two separate requirements:

For code, a verifier might be a set of unit tests. For formal proofs, it could be a proof checker; for other tasks, it might be majority voting or an LLM-based judge. The model’s outputs vary from sample to sample, and sampling temperature can influence that diversity. Azalia Mirhoseini stresses that this is not exhaustive traversal of all possible answers; that space is far too large. It is a comparatively small number of model-generated candidates drawn from a vast space.

Large Language Monkeys measured coverage, or pass@K, as the number of samples rose on MiniF2F-MATH, CodeContests, MATH, and GSM8K. It compared Llama-3 8B and 70B instruction-tuned models with a single GPT-4o attempt. Across the reported benchmarks, the Llama models began below GPT-4o’s one-attempt result but rose past it as sampling increased. The experiments increased samples per problem from one to 10,000; for some hard problems, Mirhoseini says, only three or four of those 10,000 solutions were correct.

The conclusion she draws is that a model may contain more problem-solving ability than is visible in one response; inference-time search can extract it. The model is fixed. What changes is the amount and structure of computation used to generate and select among its outputs.

That result does not make inference compute free. A student pressed on latency, and Mirhoseini acknowledged the cost tradeoff. Parallel samples can reduce latency relative to sequential generation because they can run simultaneously, but they still consume compute, and the appropriate tradeoff depends on the task’s type and complexity. Nor can diversity be increased indefinitely by raising temperature: beyond roughly 1.2, she said, outputs tend toward gibberish. Sampling and other diversity mechanisms must be tuned so that added search produces useful variation rather than noise.

OpenAI’s o1 results add a related form of evidence: pass@1 accuracy on difficult AIME math problems rises with both train-time and test-time compute. Large Language Monkeys demonstrated increasing coverage when an oracle-like verifier could select from many candidates; the o1 framing emphasizes increasing the chance that the model’s final single answer is correct.

? aakanksha-chowdhery characterizes the distinction as movement from pass@KThe chance that at least one answer among K samples is correct. or coverage toward pass@1The chance that the model's single final answer is correct.: the system learns which reasoning paths are correct, rather than merely having a correct path somewhere in its sample distribution.

Repeated sampling is easiest where correctness is mechanically checkable—math, code, and formal domains. In less verifiable domains, the field turns to LLM judges, learned reward functions, tools, and other feedback systems. Generating many candidates is useful only if a system can distinguish the useful ones.

Reasoning models try to make that search more productive. Azalia Mirhoseini describes behaviors that make extra test-time computation useful: analyzing a problem, decomposing it into simpler steps, evaluating intermediate work, correcting mistakes, and proposing alternative approaches when an initial path fails.

One o1 example asks for a bash script that transposes a matrix represented as a string. Before producing code, the model identifies the input and requested output formats, then lays out a plan: parse the string, build an array-of-arrays representation, transpose the matrix, and format the output. In a separate chemistry example, the model begins with the wrong pH calculation, stops itself with “Wait, the correct formula is,” and revises the calculation.

These traces resemble the chain-of-thought demonstrations once supplied in prompts, but the model is generating its own working process. Mirhoseini argues that this added computation is worth paying for when it produces better answers on difficult problems. The gains are not uniform: o1-preview received stronger human preference than GPT-4o in mathematical calculation, data analysis, and programming, but not necessarily in personal writing or text editing.

The model’s skills are partly bootstrapped. ? aakanksha-chowdhery says instruction-tuning data commonly includes templates and examples of ways to think, and a base model already has some capacity to reason. But the generalization is broader than a fixed set of taught sequences: a model can analyze, backtrack, or decompose novel tasks in ways not explicitly encoded as a hardcoded procedure.

How that works remains open. When students asked whether reasoning gains come from producing visible traces, the instruction to think step by step, or the reasoning data itself, Mirhoseini said that optimization has made models into more generalized thinkers. Chowdhery added that there is no single published account that fully resolves how reasoning models are taught. Outcome reward models and process reward models are among the feedback mechanisms the instructors identify as relevant.

The self-improvement loop begins when test-time search produces training material. For math, a system may generate several reasoning paths that reach a known gold answer. For code, it can generate many programs and retain outputs that pass tests. Those successful traces can become synthetic data for fine-tuning, improving future performance.

Mirhoseini connects this loop to systems such as DeepSeek-R1, OpenAI’s o-series, and Gemini Thinking: pre-training, fine-tuning, and test-time work begin to form a loop rather than a linear pipeline.

Chowdhery is more cautious about the theory behind that proposition. A student suggested that if repeated sampling already produces a correct answer, feedback should improve pass@1 but not create a dramatic underlying capability jump. Chowdhery agreed that the mechanism is not completely understood. There are different views on the relative importance of pre-training’s diverse data, reinforcement learning, and feedback-driven post-training. Both processes appear to help, but there is no single consensus on what produces the gains or how far they can continue.

The practical bottleneck is the generator–verifier gap. A model can generate a large amount of plausible, nonsensical, or partially useful content. Improvement requires a feedback loop that can decide what is actually useful. In creative writing, feedback is difficult to formalize and human evaluation becomes a limiting resource. In code, math, and rule-based tasks, tests and formal checks make improvement more tractable. Robust verification, Chowdhery says, remains one of the major constraints on making these systems better.

The transition from chatbot to agent is a transition from responding to completing a task. ? aakanksha-chowdhery defines an agent as a system that receives a goal, plans steps, acts in an environment, gathers feedback, revises its approach, and decides when to stop—or reports that it cannot complete the goal. It may use external tools and memory to remain oriented to the task.

The broader agent ideal differs from the more common agentic workflow: a predefined orchestration of language-model calls and tools. A generator may propose a solution while an evaluator accepts or rejects it with feedback. Multiple calls may investigate different lines of inquiry in parallel, after which an aggregator synthesizes an output. These structures are more constrained than an open-ended agent loop, but they are often easier to build and make reliable.

The basic patterns are prompt chaining, routing, parallelization, orchestrator-worker designs, LLM judges, and verifiers. Prompt chaining decomposes a complex goal into sequential subtasks. Routing reserves a more expensive workflow for difficult inputs. Parallelization lets independent research or problem-solving calls run at once before aggregation. An orchestrator is effectively a central LLM manager that plans and directs subsequent calls.

These components map directly onto the earlier account of inference scaling. An agent needs planning and multi-step reasoning to choose actions; tools supply information or change the environment; critics and verifiers create feedback; self-correction uses that feedback to change course.

Coding agents make the loop concrete. An agent receives an instruction, clarifies intent if necessary, searches a repository, views and edits files, executes commands in a terminal, runs tests, and continues until the relevant tests pass. Chowdhery says the broad architecture has not radically changed from earlier agent systems. Reliability has improved principally because models are stronger, reinforcement learning has improved, verifiable rewards work better, and test-time scaling can be applied more effectively.

Coding is especially promising because it combines repetitive work with inspectable state and observable feedback. Chowdhery names code migrations, version upgrades, refactors, data cleaning, ETL development, data-warehouse migrations, and unit-test generation as examples. An agent can even generate tests for code it writes, though the reliability of generated tests itself becomes part of the verification problem.

Other applications retain the same pattern but weaken the feedback signal. Customer-support systems can transcribe calls, retrieve knowledge-base material, suggest replies, summarize conversations, update tickets, and potentially take actions such as issuing refunds. Research agents can identify references, construct an outline, synthesize sources, and draft a report. AI-scientist systems extend the workflow to idea generation, experiment iteration, result processing, and manuscript writing.

Chowdhery does not present those systems as free of hallucination. Their value can be in generating a wider set of ideas or reducing repetitive work, but their outputs still need evaluation. That returns agents to the central constraint: open-ended task completion is not simply a matter of adding more model calls. It depends on whether the system can obtain feedback robust enough to know that a plan, an action, or a final result is actually good.