Exponential Families Link Likelihood, Inference, and Multiclass Classification

? chris-re presents the exponential family as a distributional framework for supervised learning: choose a probability model appropriate to the type of target, and that choice gives a common way to describe likelihood, inference, and learning. The framework itself does not require a linear model. Linearity enters later, when a generalized linear model (GLM) makes the distribution’s natural parameter a linear function of features.

The family has the form:

The notation separates three roles. $T(y)$ is the sufficient statistic: the quantity extracted from an observation. In the course’s main examples, it is simply $T(y)=y$. $b(y)$, the base measure, can depend on the observed outcome but not on the natural parameter $\eta$. The remaining term, $a(\eta )$, is the log-partition function: it depends on the parameter but not on the particular observed outcome.

The critical structural constraint is the linear interaction ${\eta }^{T}T(y)$. The base measure and log-partition function may be nonlinear, but the natural parameter enters the data-dependent part of the exponent linearly. That is the form that lets apparently different distributions share a common treatment.

For a discrete outcome, normalization determines the log-partition function:

The sum ranges over every possible outcome of $y$. Ré describes it as collecting the model’s “possible worlds”: all outcomes that must be accounted for to make total probability equal one. The normalization is not just bookkeeping. It is the source of the expectation, variance, and optimization properties developed later.

The lecture places several familiar target types in this one framework:

Ré’s practical claim is not that every practitioner needs to derive these distributions before using a library. Libraries package softmax, likelihood losses, automatic differentiation, and numerically stable computation. The point of the structure is to make clear what a model predicts, what likelihood it is maximizing, and what statistical behavior its implementation should have.

The payoff of exponential-family form is that the log-partition function contains quantities otherwise derived separately for each distribution. Its first derivative gives the expected sufficient statistic:

Its second derivative gives a variance in the scalar case, or a covariance matrix in the vector case:

Because covariance matrices are positive semidefinite, $a(\eta )$ is convex. Ré uses that fact to explain why the classical GLM setting has clean optimization behavior: the log-partition function has a favorable geometry, and the resulting learning procedures can be handled with gradient methods. The lecture does not claim that every objective connected to an exponential-family model, or every broader machine-learning system using one, is automatically convex.

This is the substance behind the claim that inference “comes for free.” The phrase does not mean no work is required. It means the work of deriving expectation and variance behavior has been compressed into a form that applies across distributions. Once a distribution is in exponential-family form, differentiating $a(\eta )$ yields its expectation and variance structure.

The fixed-variance Gaussian gives a compact example. For ${\sigma }^2=1$,

can be rearranged into exponential-family form with:

and therefore:

The Bernoulli example makes a complementary point. Its natural parameter is not its ordinary probability. Starting from:

the natural parameter becomes the log-odds:

Inverting that relationship gives the familiar logistic function:

The Gaussian model associated with least squares and the Bernoulli model associated with logistic regression are therefore not unrelated gradient-based methods. They encode different distributions for the variation left unexplained by a predictor, within the same likelihood structure.

Ré briefly relates $a(\eta )$ to a cumulant generating function. Its first derivative gives a mean, its second gives a variance, and higher derivatives give higher-order cumulants rather than ordinary moments. The operational point is the lower-order structure: the distribution packages a predicted quantity and uncertainty around it together.

A GLM begins with an input vector $x\in {\mathbb{R}}^{d+1}$ and target $y$. The target type determines a conditional distribution. The GLM then makes that distribution’s natural parameter a linear function of the features:

That equation is a modeling choice layered on top of exponential-family form. $\theta$ is the learned weight vector. Given a particular input $x$, the dot product produces $\eta$, the natural parameter of the output distribution. A response or link function maps the natural parameter to a conventional distributional parameter: a Bernoulli probability $\varphi$, a Gaussian mean $\mu$, or a Poisson rate $\lambda$.

For logistic regression, the path is:

For the fixed-variance Gaussian, the map is the identity:

In Ré’s setup, inference means taking the conditional expectation:

For a Gaussian output, this is the predicted mean. For binary logistic regression, it is the probability of the positive class. For multiclass classification, it becomes a vector of class probabilities.

Learning is maximum likelihood:

Under the IID assumption, the likelihood of the training set decomposes into a product of per-example probabilities, and its log becomes a sum. Ré identifies that decomposition as the operative role of IID. Without independence, the likelihood would not separate into the same per-example objective terms. The identical-distribution assumption also means the setup is not switching among different output distributions from example to example, unless that switching is itself modeled.

For the exponential-family GLMs discussed in the lecture, stochastic gradient ascent has the recurring update:

The update is residual times features, scaled by a learning rate. Least squares and logistic regression are presented as instances of a broader procedure: select a distribution for the target, use its conditional expectation as a prediction, and optimize the associated likelihood.

Ré frames the distributional choice as error modeling. A predictor cannot account for every variation in observed data: measurement error, unmodeled noise, count variation, or ambiguity in a discrete label remain. Choosing Bernoulli, Gaussian, multinomial, or Poisson states how that remaining variation is represented. It is part of the model, not an afterthought.

For $k$-class classification, labels are encoded as one-hot vectors. A cat, dog, car, or bus label can be represented by a vector with one coordinate equal to one and every other coordinate equal to zero. A prediction is not itself a one-hot label; it is a distribution over possible labels.

Each class $j$ has a weight vector ${\theta }_j$ with the same dimension as the input. The model computes one linear score per class, then applies softmax:

Exponentiation makes each unnormalized score positive. Dividing by their sum produces values that add to one. Softmax turns competing scores into a probability allocation rather than simply declaring a winner.

The lecture’s numerical example begins with three raw scores:

Exponentiation gives approximately $[7.39,2.72,1.11]$. Normalizing by their total, $11.22$, gives:

The highest score receives the greatest probability, while lower-scoring classes remain part of the model’s output and training objective. Ré emphasizes that softmax varies smoothly across alternatives instead of immediately collapsing to a hard winner-take-all decision.

The geometry offers another way to read the scores. Each ${\theta }_j$ defines a linear scoring direction in feature space. The set of points satisfying ${\theta }_j^{T}x=0$ is a hyperplane; moving in the direction of ${\theta }_j$ raises that class’s score. A two-dimensional sketch represents these surfaces as lines separating regions that are more cat-like, dog-like, or car-like. Ambiguous regions are precisely where scores must be compared against one another.

A low-dimensional sketch is not a promise of perfect classification. Ré notes that a simple linear separator cannot cleanly distinguish arbitrary arrangements of points. His point is that learned representations can have many dimensions, where useful separation may emerge; he cites the 768-dimensional token representations of the first GPT-2 model as an illustration. Later course material, he says, introduces kernels that can expand the effective feature space further.

There are only $k-1$ independent probabilities in a $k$-class distribution because the probabilities must sum to one. When $k=2$, softmax reduces exactly to logistic regression. Beginning with:

and dividing numerator and denominator by $e^{{\theta }_2^{T}x}$ gives:

Two-class softmax is logistic regression with ${\theta }_1-{\theta }_2$ as the effective parameter vector.

The same operation appears at the output of language models, where the competing classes are tokens rather than cats or cars. Ré says the final step considers the tokens in a model’s vocabulary through softmax. A system can select the highest-probability token, but it can also sample from the resulting distribution. That distinction explains why repeated generations need not be identical.

Softmax supplies a predicted distribution $\hat{p}$. Maximum likelihood supplies the learning objective. With a target distribution $p$, the resulting loss is cross-entropy:

For an ordinary one-hot label, only one term survives. If the observed class is $i$, the loss is:

Minimizing this loss is equivalent to maximizing the probability assigned to the observed class. But competing classes remain part of training through the softmax denominator. Raising the correct class’s probability means raising its score relative to the alternatives, not simply increasing an isolated scalar.

One-hot labels make an absolute claim: the observed class has probability one and every other class has probability zero. Label smoothing deliberately weakens that claim. With three classes, the target:

can become:

That target distribution is substituted directly into cross-entropy. Rather than optimizing only the log probability of the observed class, the model receives a weighted objective involving all class probabilities.

Ré offers two motivations. Labels may be wrong or incomplete, so treating every annotation as absolute can be unjustified. And training repeatedly on particular examples can drive a model toward an overly specific version of their featurization. Adding a small amount of uncertainty acts as regularization: it discourages perfectly confident predictions on every training point.

The added mass need not be uniform if stronger information exists. A target distribution could encode that an example plausibly belongs to either a cat or dog class, while being unlikely to be a car. That distribution can be used directly in cross-entropy. The common uniform version is simpler: it introduces small uncertainty across alternatives and can make training less sensitive to imperfect labels.