Orply.

On-Policy Distillation Uses a Model’s Own Attempts to Improve Reasoning

Shervine AmidiStanford OnlineWednesday, November 18, 20263 min read

Stanford adjunct professors Afshine and Shervine Amidi argue that improving language models increasingly depends on post-training, not simply making them larger. They single out on-policy distillation, in which a student model generates its own responses and a stronger teacher evaluates those attempts. The Amidis say this makes training more closely reflect the model’s behavior in practice, especially for reasoning tasks where intermediate steps, mistakes and corrections matter.

Post-training shifts the question from scale to behavior

For a long time, much of AI progress came from pretraining: training larger models on more data. Shervine Amidi says many of the most interesting improvements now happen after pretraining. The question is no longer only how to make a model bigger. ? afshine-amidi asks how to make it reason better, follow instructions better, solve harder problems, and adapt to specialized domains.

The Amidis place finetuning, reinforcement learning, preference optimization, verifier-guided training, and distillation among the techniques used after pretraining. They single out on-policy distillation. In standard distillation, a smaller student learns from examples produced by a stronger teacher. In the on-policy version, the student generates its own responses during training, and the teacher evaluates those attempts.

In standard distillation, a smaller model often learns from examples produced by a stronger teacher. But in on-policy distillation, the student generates its own responses during training and learns from how a stronger teacher evaluates those exact attempts.
? afshine-amidi · Source

The difference is whose output supplies the example. In standard distillation, the teacher produces an answer and the student learns from that output. In on-policy distillation, the student first produces an attempt; the teacher’s evaluation is then directed at that attempt. The student is not simply being shown what the teacher would say. Its own response is the material being evaluated.

This changes the relationship between the model’s training examples and its behavior during training. The student-generated attempt makes the student’s own output central to the process, while the teacher remains involved as the evaluator. That distinction is the basis for the speakers’ claim that on-policy distillation aligns learning with how the model behaves in practice.

For reasoning, the path matters as well as the answer

Shervine Amidi says the alignment matters especially for reasoning tasks, where intermediate steps, mistakes, and corrections can shape the result. The contrast between the two forms of distillation helps explain why: the student’s own attempts are evaluated in the on-policy setup, so the focus is not limited to examples first produced by the teacher.

The point is not that the student’s attempt is necessarily correct, or that having a teacher evaluate it guarantees better reasoning. Rather, the student’s behavior during training becomes part of what the method addresses. For reasoning, that matters because a response includes more than its endpoint: the steps taken, errors made, and corrections attempted are part of the process the speakers say can matter.

This is the practical significance the Amidis attach to on-policy distillation. Standard distillation gives the student teacher-produced examples; on-policy distillation puts the student’s own responses before the teacher for evaluation. They describe that shift as a way to bring training closer to the model’s behavior in practice, particularly when the reasoning process contributes to the outcome.

The broader change they describe is a move beyond treating scale as the only route to improvement. The listed post-training techniques address what a model does after pretraining, from following instructions to adapting to specialized domains. On-policy distillation is one example of that approach: it retains a stronger teacher’s role while making the student’s own attempts the focus of evaluation.

The frontier, in your inbox tomorrow at 08:00.

Sign up free. Pick the industry Briefs you want. Tomorrow morning, they land. No credit card.

Sign up free