Long Tool-Use Trajectories Expose Self-Distillation’s Stability Limits

Ronak Malde frames continual learning as the next source of model improvement because the usual route—larger pretraining corpora followed by increasingly difficult benchmarks—is becoming both slower and less representative of real use. Benchmarks have saturated on progressively shorter timelines, he says, while the tasks used to measure frontier models are stretching from hours to days. Building and operating the environments needed for those tasks is expensive. More importantly, curated benchmark distributions may not resemble the states an AI system encounters in production.

The alternative signal is already being generated. Malde estimates that production AI systems consume hundreds of trillions of inference tokens each day. Those interactions contain evidence of where models helped, failed, took wrong turns, or required intervention. In his account, that stream is largely discarded rather than used to update the systems that produced it.

The aspiration is not merely to retrain occasionally on user data. It is a loop in which deployed agents generate traces, traces inform an update, and the updated system is deployed again. Malde compares this to the way humans learn from real-world experience rather than from a fixed curriculum. But he also makes clear in the discussion that this is not yet what current systems achieve: most work remains “pseudo-continual learning,” with offline batch updates followed by redeployment.

The obstacle, as he presents it, is not a lack of interest in continual learning but a mismatch between the properties of present-day post-training methods and the properties a production learning loop would require. He evaluates methods on four criteria:

The claim is that each familiar method leaves at least one of these axes offline or coarse. Supervised fine-tuning has a single training example and per-token supervision, but learns from fixed datasets rather than its current production behavior. DPO and RLHF move closer to online task distributions, in Malde’s telling, but still use off-policy samples, paired comparisons, and sequence-level rewards. GRPO restores on-policy rollout generation, which he credits with much of the power of contemporary reinforcement learning, but does so by accepting a costly bargain: offline task distributions, multiple parallel rollouts, and one trajectory-level reward.

That last limitation is more than an implementation detail. Malde’s analogy is an essay returned with “87 out of 100” and no comments. The grade ranks the result but does not explain how to improve its individual sentences. A trajectory-level reward can similarly identify winning and losing rollouts, but it gives the policy only a compressed signal about the decisions that led there.

Ronak Malde proposes on-policy self-distillation, or OPSD, as an answer. The algorithm begins from ordinary knowledge distillation: a smaller student model and a stronger teacher see the same fixed data, and the student is trained to match the teacher’s token-level log probabilities. On-policy distillation makes one important change: the data is no longer a fixed dataset. The student first generates a trajectory, and both student and teacher are evaluated on that trajectory.

The frontier problem is immediate. If the model being trained is already the strongest available model, there may be no smarter teacher from which to distill. Self-distillation substitutes privileged information for a stronger model. The same model acts as teacher after receiving a hint—some additional information about the task or environment that the student did not receive. Training then matches the ordinary student’s log probabilities to those of the hinted version.

A simple illustration is a derivative problem. The student receives the problem alone. The teacher receives the same problem plus a reference solution showing the derivative and its evaluation. The student is not asked merely to copy a final answer; it is trained toward the token distribution the model would have assigned had it possessed that additional guidance.

In Malde’s formulation, this gives OPSD all four desired properties. A system can learn from online interactions rather than a benchmark distribution. Its samples are generated by the current policy. It needs only one rollout rather than a group of alternatives. And instead of receiving one score for a full trajectory, it matches log probabilities at every token.

The dense target changes what is optimized. In GRPO, Malde says, an update primarily sharpens or suppresses the tokens that appeared in sampled rollouts according to their relative reward. In OPSD, the teacher supplies a distribution over the full vocabulary at each position, so a token the student did not sample can still receive increased probability if the hinted teacher assigns it higher probability.

For a vocabulary of roughly 65,000 tokens, this lets OPSD shift the distribution toward alternatives that were plausible under the teacher’s privileged view but absent from the student rollout, rather than only reinforcing sampled choices. Malde argues that this is why the method can move beyond the plateau he observed with GRPO on short-horizon coding tasks.

In Malde’s displayed LiveCodeBench v6 comparison, SDPO sits above GRPO and the Claude Sonnet 4 reference line. A second comparison labels SDPO with an “8.1x speedup” relative to GRPO during training; a third shows shorter responses and labels the result a 0.4x reduction in total cost. His explanation is that RL often improves by allowing models to “think” for more tokens, whereas OPSD can improve while the tokens needed to solve difficult tasks decline.

For short-horizon applications, he says, this can already be used in relatively direct ways: chatbots that learn from unstructured user behavior, for example, using open-source tooling such as OpenClaw-RL. The harder question is whether the method remains stable once trajectories become long, tools are involved, and models are large.

Ronak Malde says Trajectory’s scaling work exposed failures that do not show up in short-horizon demonstrations. The company has tested the approach at 120-billion, 500-billion, and one-trillion-parameter scales. At roughly 120B parameters, with agents making 50 to 100 tool calls rather than one or two, he reports unstable evaluation accuracy, substantial run-to-run variance, and tool-call errors. The model can fail to behave according to the format it learned during instruction fine-tuning.

The first failure is what Malde calls the “but wait” problem. On a short task, student and teacher can remain close enough that matching their distributions is useful. On a long task, the student’s on-policy trajectory can drift far from the trajectory the hinted teacher would naturally produce. Conditioned on the student’s increasingly mistaken prefix, the teacher tries to redirect the response at every available token.

The result is not a clean correction. Malde says terms such as “wait,” “but,” and “maybe” become increasingly prominent as OPSD training proceeds. He characterizes the endpoint as a local, suboptimal equilibrium in which the model is caught between two divergent distributions and “everything just turns into maybe.”

His proposed fix is step-level divergence weighting. In a tool-calling trajectory, the system measures the divergence between student and teacher distributions over each step, rather than treating the entire trajectory as equally trustworthy. That divergence becomes a multiplier on the token-level training weight—not simply a KL penalty.

The practical effect is selective correction. If an early tool-call step is in distribution, it can retain its ordinary weight. If a later step diverges heavily, the algorithm can focus on repairing the first bad step rather than applying strong corrections across everything downstream. Because weights are independent by step, a trajectory that goes off course can still be useful if it returns to a reasonable state later. Malde describes this as allowing both error suppression and error recovery: reduce pressure on badly divergent portions without throwing away later segments that have come back into alignment.

This qualifies the original promise of per-token supervision. Dense feedback is useful only when the teacher’s distribution represents a learnable correction, not when it reflects privileged knowledge of a radically different trajectory.

Ronak Malde identifies a second long-horizon problem: information leakage. Reinforcement learning has reward hacking, in which a model finds ways to optimize the score without doing the intended task. OPSD has an analogous failure, he argues, when the teacher’s hint contains information the student could never have known from its own state.

His example is a mathematical problem asking for the final three digits of a large expression. If the privileged hint contains both the solution method and the answer—“000”—the model may generate a reasoning trace that begins as if it were independently solving the problem, then incorporates the known answer and reconstructs a plausible explanation around it. That trajectory is not a skill the deployed student can reproduce. It is a shortcut enabled by teacher-only information.

The straightforward mitigation is filtering. An LLM can rewrite a revealing hint into a nudge that points toward relevant evidence without handing over the conclusion. In a support example, the raw hint says that a user’s SSO token is expired and tells the agent to search logs to find it. The filtered version says only to search the logs. The distinction is between identifying the location of evidence and supplying the answer before the agent has found it.

Malde says that works “decently well,” but presents residual guidance as a more algorithmic response. It compares a teacher with a partial hint against one with the full hint. The partial-hint teacher represents a less privileged view; the gap between its distribution and the full-hint teacher’s distribution indicates how strongly the full hint is pulling the target away from what the unhinted student could plausibly infer.

The training target is a linear combination of the two distributions rather than a full match to the fully hinted teacher. If the partial hint remains near the student’s distribution while the full hint does not, the combined target limits the shift into inaccessible territory. The point is not to eliminate privileged information—the hint creates the teacher—but to separate guidance that teaches a transferable next step from guidance that exposes an answer.

With these modifications, Malde says Trajectory has trained a 120B model on Mercor APEX-Agent tasks that often require more than 100 tool calls. Trajectory’s slide reports mean reward of 3% for the baseline, 18% for SDPO, and 21% for “SDPO++,” described as the core algorithm with stability-oriented tweaks for larger models and long-horizon tasks.

Ronak Malde does not present OPSD as a complete solution to continual learning. He calls it a substantial step because it preserves the on-policy behavior that makes RL valuable while adding online task distributions, single-rollout training, and dense token-level feedback. But the operational loop remains unresolved.

Trajectory’s product thesis is to capture agent traces from production, optimize them through a self-serve process, deploy updated systems, and govern that cycle through a control plane. The intended payoff is not simply a model that is periodically retrained, but an agent and its surrounding system that improve with use.

The final discussion sharpens the caveat. Asked whether continual learning means low-latency, real-time updates or batch processes, Malde says current work still lies in a middle ground. Systems can collect data and update offline, then “re-upload” a model, but the infrastructure and algorithms for a truly continuous loop are not yet in place. A product with 10,000 active rollouts must still merge and use those experiences safely and coherently.

He also identifies the agent harness as an underexplored part of the problem. Improving the model alone is not the only learning target; the harness—the surrounding structure that directs tool use and task execution—can improve too. The more consequential question, he says, is how model and harness change together while both are being updated online.