PPO, GRPO, and Distillation Trade Off Credit Assignment and Training Cost

In post-training, reinforcement learning changes a model’s parameters to make high-reward responses more likely. The setup is familiar from ordinary reinforcement learning: an agent observes a state, takes an action according to a policy, and receives a reward. For a language model, the state can be the input so far, the action the next token, and the policy the distribution over possible next tokens. Because assigning a meaningful reward to each token is difficult, training often treats the prompt as the input, the complete response as the action, and the response-level score as the reward.

Shervine Amidi used two examples to distinguish the kinds of reward that can guide this process. In preference tuning, there may be no objectively correct answer: people might prefer one response’s tone over another’s. Those preferences can be modeled with a reward model trained from pairwise judgments, using the Bradley–Terry formulation. This is the setting commonly called reinforcement learning from human feedback, or RLHF. Human judgments train the reward model; once trained, the reward model can score later generations without a human judging every response in the training loop.

In reasoning or coding tasks, by contrast, an answer may be checkable. A math solution can be compared with a known answer; code can be tested against test cases. These are examples of reinforcement learning with verifiable rewards, or RLVR. The reward signal differs by setting: in one, a model approximates subjective preferences; in the other, a deterministic check can indicate whether an answer is correct. In either case, the reward tells the training procedure something about the completed response. It does not necessarily explain which words or steps made the response good or bad.

The basic objective is to maximize the expected reward of trajectories sampled from the current policy. For an LLM, that expectation can be understood as an average over prompts and then over responses generated for those prompts. The gradient matters because it indicates which parameter changes will increase the objective. The derivation rests on a useful identity: the gradient of a function can be written as the function multiplied by the gradient of its logarithm. Applying that identity to the policy yields the policy-gradient expression:

The practical intuition is direct. A sampled response with positive reward pushes the model toward increasing its probability; a response with negative reward pushes it toward decreasing its probability. The reward weights the direction indicated by the log-probability gradient. This is the basis of REINFORCE, which the instructors presented as the simplest Monte Carlo version of policy-gradient training: given a prompt, sample a response, score it, compute the reward-weighted gradient of its log probability, and update the parameters with gradient ascent.

The objective is an expectation, so in practice the training procedure estimates it from sampled responses. A single sample gives a noisy estimate; a batch gives several samples to average. But each sample must first be generated by the model. Unlike pre-training or supervised fine-tuning on fixed examples, this procedure needs fresh generations from the policy being trained. Generation is sequential: the model produces a token, uses it as context to produce the next token, and repeats. This makes sampling a response expensive.

Evaluating a response that is already known is different. Given the full sequence, the model can process its tokens together and compute the corresponding probabilities in parallel, subject to its causal attention pattern. That evaluation is much cheaper than producing the sequence token by token. This difference makes it possible to reuse a batch of sampled responses for more than one update, rather than generating a new batch for every parameter update.

The reuse is expressed through an importance-sampling ratio. Instead of taking an expectation over responses sampled from the current policy, one can take an expectation over responses sampled from a fixed, older policy and weight each response by the ratio between its probability under the current policy and its probability under the old one. The ratio corrects for the fact that the data came from the old policy. It does not make generation free; it makes it possible to generate once and reuse the resulting examples across updates.

The instructors emphasized notation because papers use it inconsistently. In this lecture, the policy ratio is written as ρ to distinguish it from a reward, which is often also written as r. Likewise, π_old may refer to the policy that generated the latest batch, whereas π_ref typically means a fixed reference policy. And an expression labeled L may be an objective to maximize rather than a loss to minimize. The direction of optimization, and the policy represented by each subscript, have to be read rather than assumed.

The first problem with the basic reward-weighted gradient is variance. If one answer receives reward 101 and another receives reward 100, both produce large updates, even though one is only marginally better than the other. The gradient uses the absolute score, not the answer’s performance relative to an appropriate baseline. In a toy example where the two answers are equally likely, the instructors showed how subtracting a baseline near 100.5 changes the effective signals from large values such as +50.5 and −50 to much smaller, more comparable values. A baseline makes the update reflect whether an outcome did better or worse than expected, rather than simply whether its raw reward was large.

The second weakness is that a completion-level reward does not say which tokens deserve credit. Consider the prompt: “Teddy is in a sealed box. The box moves to the kitchen. Where is Teddy?” A response might say that Teddy climbs out of the box, walks to the kitchen, and is therefore in the kitchen. Its final answer is correct, but the reasoning is not: Teddy remained in the sealed box while the box moved. With a single reward for the completed response, the basic method would reinforce all of its tokens together. A correct final answer does not imply that every intermediate step was useful.

The third weakness is control over how far the policy moves. The starting model has already been pre-trained and supervised-fine-tuned; it is not an empty system. A post-training objective may improve one target while eroding earlier capabilities. And the reward itself may be an imperfect proxy for what is wanted. The instructors illustrated reward hacking with a lecturer optimizing applause: if applause alone determines success, the lecturer could spend the class telling jokes rather than teaching. A model trained against an imperfect reward can similarly learn to produce what scores well without producing what the reward was intended to represent.

PPO—Proximal Policy Optimization—addresses these problems by replacing raw rewards with advantages and by limiting policy movement. The distinction between reward, value, and advantage is central. Reward is the score assigned to an outcome (or, in some formulations, to a step). Value is an estimate of expected future reward from a particular state or prefix. Advantage compares an action’s contribution with what would have been expected from that point. Reward asks, “How good was this?” Advantage asks, “How much better or worse was this than the baseline?”

The baseline in PPO is supplied by a value function. Given a prompt and a partial response, the value function estimates the expected future reward if generation continues from that point under the policy. In the Teddy example, suppose the expected final reward after the prefix “In” is 0.6, after “In the” it is 0.7, and after “In the box” it is 0.95, with the completed answer reaching 1.0. The token “box” makes a positive contribution relative to the preceding prefix: the estimate rises from 0.7 to 0.95. In the simplest illustration, the advantage of a token is the value after it is added minus the value before it.

That simple difference is an intuition for token-level credit, not a complete description of PPO’s advantage calculation. A token can affect not only the immediate estimate of reward but also the prospects of later tokens. A token that initially seems unremarkable may make a good continuation possible; conversely, it may lead the response onto a path that makes later reward less likely. The instructors pointed to Generalized Advantage Estimation (GAE) as a way to combine information about rewards and future values when assigning credit through a sequence. They did not develop GAE in detail, but stressed that it is more involved than merely subtracting two consecutive value estimates.

The value function estimates future returns at prefixes. The lecture described it as trained jointly with the policy, using a regression objective to predict the eventual completion-level reward from partial completions. Its predictions are defined at the token level: for each prefix, it estimates the reward expected if generation continues. Where a reward model is used, that model is trained beforehand and frozen; the value function is trained during policy optimization because its estimates concern continuations under the policy being trained.

The instructors described the policy and value functions as distinct, while noting that they may share a backbone. The value function does not feed into the policy as an input. Rather, the policy and value losses can be trained jointly, with a value-prediction objective alongside the policy objective. The exact model arrangement is therefore not simply that PPO always requires two fully separate networks: the important point in the lecture is that the value function is an additional learned function, trained in coordination with the policy.

PPO also needs to account for batch reuse. Responses are sampled under the old policy, then evaluated and used for updates to the current policy. At a token or action, the policy ratio compares the current policy’s probability with the probability assigned by the policy that generated the batch. Multiplying the advantage by this ratio corrects for the fact that the batch came from the old policy rather than the current one. The ratio is not the reward; it is a probability correction.

The correction alone does not address the concern that updates may move the policy too far. PPO’s clipped objective limits the ratio’s contribution to the objective around one, conventionally described by lower and upper bounds of one minus epsilon and one plus epsilon. If an action has positive advantage, increasing its probability initially improves the objective. Once the ratio crosses the upper clipping boundary, the objective stops rewarding further increases in that direction. For an action with negative advantage, the corresponding incentive to reduce its probability is capped at the lower boundary. In both cases, clipping limits the objective’s incentive to keep pushing the probability in the same direction.

Clipping constrains changes relative to the old policy, the one that generated the batch. It is not the same as a constraint relative to the model from which post-training began. The instructors distinguished three policies that can appear in PPO formulations:

• π_θ is the current policy being updated.
• π_old is the policy that generated the batch used for the current set of updates.
• π_ref is a reference policy, often the supervised-fine-tuned model, that remains fixed.

The old policy changes as training proceeds: after a new batch is collected, that batch-generating policy becomes the old policy for the next update cycle. The reference policy, by contrast, stays fixed. Clipping constrains changes relative to the batch-generating policy; a KL penalty constrains changes relative to the reference. The two comparisons address different concerns and may be used together.

The KL penalty measures divergence between probability distributions. In the objective shown in the lecture, a coefficient β scales the penalty between the current policy and the reference policy. A larger coefficient makes deviation from the reference more costly. The instructors connected this coefficient to the one in Direct Preference Optimization (DPO), where the reward is expressed using a policy-to-reference log-probability ratio. They also noted a practical limitation: computing KL can be expensive because it involves the whole vocabulary, so implementations use estimators.

The lecture’s DPO connection was about the role of the reference policy and the coefficient, not a claim that DPO and PPO are the same training procedure. In the DPO formulation discussed, a preference objective compares the relative log-probability ratios of a winning and losing response against the reference policy. Those ratios define a reward-like quantity. The coefficient controls how strongly the policy’s relationship to the reference enters that quantity. In PPO, a KL term can directly penalize the current policy for moving from the reference. In both cases, the reference distribution matters, and the scale coefficient affects the strength of that relationship.

The exact placement of the KL term varies. Some formulations place it directly in the objective, while others incorporate its effect into the reward or advantage calculation. The lecture’s comparison of PPO and GRPO makes this implementation variation explicit: the location of the penalty can differ even when the method is intended to restrain divergence from the reference. The distinction between a penalty against the reference and clipping against the old policy remains useful even when a particular implementation combines terms differently.

PPO combines a relative signal—advantages—with constraints on movement. The value function supplies a baseline and a token-level signal; clipping and KL penalties restrain changes to the policy, but with respect to different comparison policies. The benefit comes with a substantial operational cost: PPO requires training a value function alongside the policy. Since the value function estimates returns under the policy as it changes, the two must be trained in coordination. That additional learned function and training burden motivates a method that retains relative advantages without learning a separate value function.

Group Relative Policy Optimization (GRPO) removes the value function by estimating a baseline from a group of responses to the same prompt. Instead of generating one response and estimating its expected value with a learned critic, the model generates several responses. Each receives a reward, and each reward is compared with the group’s mean and scaled by the group’s standard deviation. The resulting score estimates how well that response performed relative to its peers.

For a prompt x, let the policy generate G outputs, each with reward R(x,o_i). The group-relative advantage shown in the lecture is:

A response above the group mean receives a positive relative score; one below it receives a negative score. This reduces dependence on raw reward scale and provides a group-based baseline without a trained value function. As with PPO, the objective can use a ratio between the current and old policies and clipping to restrain updates. The formulation shown also includes a KL penalty relative to the reference policy. In reasoning, math, or coding tasks, the reward model is optional when correctness can be checked directly.

The objective comparison presented in the lecture can be summarized by what each method uses to score and update a response:

The central distinction is the source and granularity of the advantage. In the common GRPO outcome-supervision setup described in the lecture, the group computation produces a score for each completion. That score is then applied across the tokens of that completion. PPO, by contrast, uses the value function to estimate expected returns at successive prefixes and can compute token-level advantages. The GRPO formula may write the advantage with both an output index and a token index, but the instructors noted that the usual outcome-supervised form uses the same completion-level advantage at each token. A process-supervised version would require a way to distinguish the contributions of individual steps or tokens.

The training machinery makes the tradeoff visible. GRPO uses a policy model to produce several completions, a reward source to score them, and a group computation to normalize the scores. The reference model is fixed. PPO also uses a reward source and a fixed reference model, but adds a value function trained jointly with the policy to produce token-level advantage estimates. GRPO replaces that learned baseline with comparisons among sampled outputs. It removes one trained function, but it also gives up the value function’s token-by-token estimate.

This is an economy, not a complete solution to the credit-assignment problem. Return to Teddy in the sealed box. Across several responses, one may give correct intermediate reasoning and the right answer; another may reason incorrectly but arrive at the same answer; a third may have a wrong answer despite a sound first step. A single outcome score cannot distinguish the good and bad steps inside each response. The group comparison tells the model which whole completion did better than the others, not which token or reasoning step caused the difference.

The instructors described this as locally noisy and slow to resolve. Over many groups, the model may learn which steps tend to accompany good answers, but it is not using the available feedback at each step. One alternative is process supervision, where advantages or labels are assigned at the level of reasoning steps. That requires separate labeling, however, and the ambiguity can simply move down a level: in “Teddy walks to the kitchen,” is the error that Teddy walks at all, or that he walks to that location? Finer-grained supervision can demand more detailed labels or a reward function trained to provide them. The instructors also noted that one motivation for GRPO was to avoid the burden of training additional models; adding a dedicated process reward model can bring some of that burden back.

The lecture also examined a length effect associated with the normalization in the GRPO objective. In the formulation shown, each response’s token contributions are averaged across its length, using a factor of one over response length. A token in a short response therefore receives a larger per-token weight than one in a long response. For positive advantages this makes a short response’s upward contribution larger; for negative advantages it also makes a short response’s downward contribution larger.

The instructors’ concern was the latter case. If the model is producing bad answers, a longer bad response can receive a smaller negative contribution per token than a shorter bad response. In their proposed mechanism, once the model reaches a point where it cannot improve the reward on those answers, the normalization can leave an incentive to produce longer bad responses because they are penalized less per token. They related this mechanism to a pattern shown in the lecture for DeepSeek-R1-Zero: average response length continued increasing during training, including when measured math accuracy had plateaued. The lecture offered the normalization as a possible explanation for the pattern; it did not establish that this mechanism alone caused the observed length trend.

The evidence shown was a pair of training curves attributed on the slide to DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI, 2025). One curve tracks AIME accuracy and the other average response length. The accuracy curve reaches a plateau while the response-length curve continues upward. The instructors used that contrast to motivate examining the objective’s weighting. The chart supports the observation that the two quantities moved differently over the displayed training period; the explanation that follows is their account of a possible incentive created by the normalization.

The instructors described two adjustments aimed at the normalization problem. DAPO changes the normalization so that token-level contributions are equalized across the group rather than averaging each response separately. In the displayed expression, the normalization is based on the total number of tokens across the sampled responses. Dr. GRPO, as presented, removes the response-length normalization term. These are modifications to the objective’s weighting, intended to address the particular length incentive discussed in the lecture, not evidence that every implementation or setting will exhibit the same behavior.

They also identified two other possible adjustments. First, dividing by the standard deviation of rewards may introduce a bias when prompts of different difficulty produce groups with different reward variability. One proposed response is to remove that division, retaining a group-relative difference from the mean without standardizing by the standard deviation. Second, a single clipping epsilon for both lower and upper bounds may limit exploration of low-probability actions. If an action begins with very small probability, the same upper margin may not allow it to become substantially more likely. Separate lower and upper epsilon values can make clipping asymmetric and, in the instructors’ account, better encourage diversity.

GRPO avoids PPO’s value function, but its common outcome-level form still assigns the same advantage to every token in a response. The next method aims to keep training on the model’s own generations while restoring token-level feedback—without training another reward or value function. Its proposal is to use a stronger, already available model as a teacher and compare its next-token distribution with the student’s at each point in a student-generated completion.

This is on-policy distillation. The student generates a response to a prompt. The teacher then receives the prompt and the generated tokens as context, and supplies its distribution over the next token at each position. The student is trained to bring its predictions closer to the teacher’s on those contexts. The distinction from ordinary supervised fine-tuning is that the examples are generated by the student’s current policy: the training signal concerns the student’s own outputs, including its mistakes.

The choice of KL direction matters because the divergence is not symmetric. The lecture framed KL(P||Q) as asking: in a world governed by P, how costly is it to approximate that world with Q? In ordinary distillation, the teacher is often the first distribution and the student the second. For on-policy training, the student’s generated contexts are central, so the formulation discussed uses reverse KL, KL(S||T):

The instructors described two consequences of this choice. When the teacher assigns probability to a token but the student assigns it little or none, the small student weight makes that mismatch comparatively forgiving. When the student assigns probability to a token the teacher considers unlikely, the ratio can become large and the student is penalized more heavily. This can make reverse KL “mode seeking”: the student is pushed toward high-probability regions of the teacher’s distribution rather than being required to cover every possibility the teacher represents. These are properties emphasized in the lecture’s explanation of the objective, not a claim that reverse KL is always the better choice. The instructors noted that the original work they discussed used a mixture of both directions.

The Teddy example makes the token-level behavior concrete. If the student begins a response with “Teddy,” the teacher can assess the next token in the context of that prompt. If the student continues with “climbs,” and the teacher considers that an unlikely next token, the student receives a penalty at the point where its generation diverges from the teacher’s expectations. Later tokens are judged with the earlier generated tokens included in the context. Thus, in one example, “walks” may not be penalized even though the sentence is part of a faulty explanation: once “climbs” is in the context, the teacher may regard walking as a plausible continuation. The instructors used this to underline that token-level supervision is contextual. It tends to penalize the student when its prediction first goes wrong; subsequent predictions are evaluated conditioned on the context that has already been produced.

That feedback is denser than a single outcome reward: it arrives at each generated token. It is also on-policy, because it is applied to the student’s own completions. The instructors said the method works well in practice and can be compute-efficient, including in settings where only a restricted number of parameters are updated. They cited an example in which on-policy distillation narrowed performance gaps that remained after supervised fine-tuning under constrained adaptation. They also argued that, without a reward model, there is no reward function for the student to hack; reducing the distillation loss toward zero means matching the teacher’s distribution.

Those strengths have limits. A teacher is required, and the student’s target is bounded by what the teacher can provide. The teacher’s distribution may be imperfect or fail to cover the domains the student needs. A model that is already the strongest available may have no better external teacher to learn from. The instructors suggested different uses across a model family: a faster model might benefit from distillation from a stronger counterpart, while a frontier model may rely more on methods based on verifiable labels. They also described self-distillation as a way to preserve previously acquired knowledge.

Self-distillation uses the same model as both student and teacher, but gives the teacher a hint. The hint can include a completed answer or information that leads to it. Because the teacher’s next-token probabilities are computed by supplying the context through teacher forcing, nothing prevents the model from receiving such extra information when producing the distributions used to train the student. In the example, the hint states that Teddy stays in the box throughout; the model is then asked to solve the prompt again. The resulting teacher distribution can supervise the student’s own generated response token by token. The hint is not part of the student’s answer; it is a way to elicit a teacher distribution that can serve as a training target.

A further extension uses multiple teachers. The lecture described a proposal in which domain specialists provide supervision for areas such as math, instruction following, software engineering, search, or safety, rather than relying on a single teacher for all capabilities. The motivation offered was that post-training against one teacher or one set of objectives may lead to forgetting, while different expert teachers can contribute domain-specific signals. Multi-teacher on-policy distillation combines the student’s generated tokens with teacher distributions selected for the relevant domain.

That observation captures both the strength and the limit of token-level distillation. The feedback can identify where a generated continuation first diverges from what the teacher expects, rather than assigning one undifferentiated score to a whole answer. But later tokens are judged in the context of the student’s earlier choices, including errors. The teacher signal is therefore dense and conditional, not an independent correction of every step in the response.