Zero Training Loss Does Not Eliminate the Need for Validation

? chris-re begins from a contradiction that reshaped how many machine-learning practitioners think about overfitting. The classical account says that, after a model becomes too flexible for the available data, test error should rise: a high-degree polynomial that interpolates a small sample should chase noise and behave badly on new points.

Modern deep networks routinely challenge that expectation. They can have many more parameters than training examples, fit the training data exactly, and still generalize well. Ré puts the practical puzzle plainly:

The familiar U-shaped picture of test error is therefore incomplete, not useless. It describes a classical regime in which adding capacity first lowers bias and then raises variance. But models can move beyond the point where interpolation becomes possible. At the interpolation threshold, Ré says, there may be effectively one zero-training-loss solution, and it may be unstable or “wild”: the model has just enough freedom to fit every point, but little room to choose a well-behaved fit.

Past that threshold, there can be many zero-loss solutions. The learning procedure may then select a smooth solution that generalizes better. Ré calls the resulting pattern double descent: as model capacity rises, test error can follow the conventional U-shape, climb sharply near interpolation, and then decline again in a substantially overparameterized regime.

The lecture’s double-descent chart places model capacity on the horizontal axis and test error on the vertical axis. It marks a classical regime on the left, an interpolation threshold at the central peak, and an overparameterized modern regime on the right, where test error falls again. The visual is intended as a qualification to the standard bias-variance picture, not a claim that capacity alone ensures good generalization.

That distinction makes zero training loss an ambiguous signal. A high-degree polynomial fitted to a small noisy sample can interpolate by oscillating wildly. A large neural network can also interpolate, yet occupy a regime in which optimization selects a smoother predictor. The shared training loss does not make the two procedures equivalent.

Ré’s boundary condition is important: the classical bias-variance decomposition remains correct for the fixed learning procedure being analyzed. What it does not itself specify is how gradient-based optimization in a very large model chooses among many zero-loss fits.

Ré frames generalization as a question about procedures, not merely one observed dataset. There is an unknown function that generates outcomes, but the learner never sees it directly. It sees noisy samples generated as y = h_*(x) + ε, with ε modeled as zero-mean Gaussian noise of variance σ².

A training set is one draw from a population of possible datasets. The relevant question is what happens when the same model class and fitting procedure are applied to different plausible samples.

The polynomial example makes the two canonical failures visible. If the true relationship is quadratic but the learner fits a line, no amount of resampling will let the linear class represent the curvature. Different samples can yield somewhat different lines, but the principal failure is systematic: the hypothesis class is too rigid. This is underfitting, or high bias.

At the other extreme, a high-degree polynomial can interpolate every point in a noisy training set. Training loss can fall to zero, but retraining on another sample yields a substantially different curve. At a fixed input, predictions from those curves can vary enormously because the model has followed noise that differs from sample to sample. This is overfitting, or high variance.

The observed behavior points to different operational problems. Persistent error even on training data is consistent with a model class that cannot express the relevant structure. A fit that looks excellent on its training sample but changes sharply when the sample changes is unstable. In linear regression, nearly unconstrained directions in the data can produce the same instability even when the class is not obviously extravagant.

The model that behaves well in Ré’s illustration is neither simply the smallest available model nor necessarily the model with zero training error. It has enough flexibility to represent the relevant structure while remaining stable as the training sample changes. Ré uses “inductive bias” for this match between a model class and the structure of a problem, while cautioning that the term is often used loosely. In the polynomial example, it amounts to having degrees of freedom that roughly match the true function.

The same concern has an analogue in classification, though Ré presents the formal derivation only for squared-loss regression. A classifier can draw a highly irregular decision boundary that perfectly separates training points, including small islands of one class within another. Small changes in an input can then change the predicted label. Ré does not treat every local wiggle as proof of overfitting: a genuine class boundary may be complicated, particularly near its boundary. But isolated pockets and discontinuities are warning signs that the representation may not capture the class structure or that the fit is following sample-specific artifacts.

To make the picture precise, Ré fixes a test input x. A training set S is drawn, a hypothesis h^S is trained on that set, and then a fresh noisy outcome is drawn at the same point. The quantity of interest is expected squared error over both sources of randomness: the training-set draw and the fresh noise at the test point.

He defines the average prediction across possible training sets as h^avg(x) = E_S[h^S(x)]. Under the lecture’s squared-loss regression setup and its stated independent, zero-mean test-noise model, the resulting decomposition is:

The first term, σ², is noise in the test observation. Ré gives measurement error and lag as examples of what can make it nonzero. A perfectly measured target could make this term zero, but a fitting procedure cannot eliminate error that remains in the test process.

The second term is bias squared: the gap between the true function and the average prediction produced across repeated datasets. A line fit to a quadratic has bias because, even averaged across many samples, the fitted procedure cannot recreate the curve.

The final term is variance: how much the prediction at the same input changes as the training data are redrawn. A flexible model can have little bias but high variance if each new sample supports a notably different fitted function.

This formalism does not require the selected hypothesis class to contain the true function. If the class is too small or otherwise mismatched to the data-generating relationship, that failure appears as bias. Nor is the average predictor simply the model one would obtain by fitting all available data. It is an expectation over possible training sets and their probabilities under the assumed sampling process. In some consistent procedures those objects may converge, but they are not the same by definition.

The decomposition also corrects a common shortcut. Training error is not simply bias, and test error is not simply variance. Test error includes noise, bias, and variance. Training error can expose an obviously rigid model, but an expressive model can push training loss toward zero whether or not it remains stable on new data.

Ré presents regularization as a controlled trade: increase bias modestly if doing so cuts variance enough to lower total test error. In Bayesian terms, it encodes prior beliefs about plausible parameter values. In the frequentist framing used through the lecture, it stabilizes a procedure that would otherwise react too strongly to weakly constrained directions in the data.

For linear regression, ridge adds an L₂ penalty to least squares:

At ρ = 0, ridge is ordinary least squares. As ρ becomes very large, the coefficient penalty dominates and pushes θ toward zero. The useful setting lies between those extremes: enough shrinkage to stabilize the fit without discarding too much information in the data.

The underdetermined case shows why the penalty matters. When there are fewer samples than dimensions, n < d, the matrix XᵀX is not full rank. If v belongs to the null space of X, then X(θ + v) = Xθ. Adding v changes neither training predictions nor training loss. Ordinary least squares can therefore have infinitely many parameter vectors that fit equally well. The data leave a direction of ambiguity.

In that illustration, the L₂ penalty eliminates null-space components that do not improve training fit: they increase the norm while leaving predictions unchanged. This is distinct from the more general effect of positive-ridge regression, which changes the fitted estimator rather than merely breaking ties among ordinary-least-squares solutions. Ré also notes that the Penrose pseudoinverse selects a minimum-norm solution in the underdetermined setting.

The ridge normal equations make the conditioning argument explicit:

If λᵢ is an eigenvalue of XᵀX, adding ρI shifts it to λᵢ + ρ. Since XᵀX is positive semidefinite, its eigenvalues are nonnegative; for positive ρ, the shifted matrix is invertible.

The benefit is not limited to eigenvalues that are exactly zero. Weakly determined directions with very small eigenvalues can make ordinary least squares unstable: a small change in data can produce a large change in the estimate. Ré characterizes ridge’s variance behavior in such a direction as changing from an inverse dependence roughly like 1/λᵢ to λᵢ / (λᵢ + ρ)².

Adding a small positive value can sharply reduce sensitivity in nearly singular directions. This is why regularization can lower variance even if XᵀX is technically full rank but poorly conditioned.

That principle extends beyond ridge. Ré points to weight decay, dropout, data augmentation, and data-adaptive methods as ways to discourage a model from placing too much trust in any single training point or incidental feature.

Data augmentation makes the principle concrete. Rotating or slightly zooming an image of a cat should not change its class. Training on transformed inputs encourages the model to ignore those changes rather than treating accidental pixel-level details as decisive. In Ré’s account, the common objective is robustness: make it harder for the model to fit artifacts that will not persist in new data.

A fixed benchmark creates a separate concern from ordinary training-set overfitting. Researchers repeatedly inspect results on the same test set, learn from its outcomes, and tune methods to its peculiarities. Even without deliberate cheating, model rankings might become artifacts of the benchmark rather than evidence of generalization.

Ré uses the ImageNet-to-ImageNetV2 comparison to examine that concern. The researchers rebuilt an ImageNet test set using the original collection procedure as closely as possible, with different images collected years later, then re-evaluated existing classifiers.

The lecture’s scatter plot puts original ImageNet accuracy on one axis and accuracy on the rebuilt test set on the other. Its points fall materially below the y = x line, showing that absolute accuracy did not transfer unchanged. Yet they remain approximately linear: models that performed better on ImageNet also performed better on ImageNetV2.

That combination is the important result. The models incurred a substantial absolute loss on the newer test distribution, but their relative ordering did not scramble. In Ré’s interpretation, ImageNet had retained useful comparative signal despite years of intensive benchmark-driven research and substantial incentives to improve leaderboard positions.

The result is qualified good news, not permission to disregard evaluation discipline. It does not establish that repeated benchmark tuning is harmless, and it does not make distribution shift a minor issue. Rather, it shows that in this case a separately collected test set imposed a meaningful accuracy cost without overturning which models were comparatively stronger.

Model selection is the operational form of the generalization problem. A standard setup uses three disjoint datasets:

The separation between training and development data is essential because the candidate with the best training loss is often precisely the candidate most able to overfit. If polynomial degree is selected by training and scoring on the same sample, the most expressive polynomial can win by interpolating that sample. A development set supplies separate data on which each candidate must generalize.

The final test set is reserved for the end because repeated inspection can leak information into development and inflate the assessment. A dev set lets practitioners try regularization strengths, architectures, or step sizes without consuming the final estimate of generalization.

There is a data-efficiency trade-off. Reserving a large development set leaves fewer examples for fitting, even though more training data often improves the model. K-fold cross-validation addresses that tension by partitioning the data into several folds. The learner trains on all but one fold and validates on the held-out fold, rotating the holdout role across folds. Each score is computed on data excluded from that particular fit, while the overall process uses limited data more completely.

Compute creates a separate bottleneck. Training every hyperparameter configuration to completion can be expensive when each run requires substantial time. Ré’s practical answer is successive halving, the core mechanism in Hyperband.

The procedure begins with many candidate configurations—different values of ρ, for example. All receive a small initial budget. They are scored, the worst half is discarded, and the survivors receive twice as much additional training. The process repeats until one candidate remains.

The surviving runs continue rather than restart. Each round uses roughly the same total compute, but better-scoring candidates receive exponentially more of it. Some work spent on discarded configurations is necessarily lost, but the procedure avoids allocating full training budgets to candidates that do not survive an early round.

Ré says successive halving has theoretical analysis under assumptions about score behavior. Its appeal is also operational: it is simple to implement and turns tuning into an explicit allocation problem under finite data and finite compute.