Shared Covariance Makes Gaussian Discriminant Analysis a Linear Classifier

? chris-re introduces Gaussian discriminant analysis as a reversal of the direction taken by the course’s earlier models. Linear regression, logistic regression, and generalized linear models learn a conditional relationship from features to target: $P(ymidx)$. GDA instead specifies how features arise within each class, $P(xmidy)$, alongside the class prior $P(y)$.

That distinction changes the question asked of a new observation. A discriminative classifier seeks a separating rule. A generative classifier asks which class distribution better explains the observation, after weighting that explanation by how common the class is.

The cat-and-elephant example makes the contrast deliberately stark. A discriminative model can draw a boundary in a feature space of height and weight without learning the internal structure of either population. It can devote its capacity to the region where the classes might be confused. GDA instead fits one distribution to cats, one to elephants, and a prior probability for each, then compares the resulting class scores for a new point.

The prior is part of the classification rule, not a decorative addition. If cats greatly outnumber elephants in the training data, an observation near the apparent boundary needs stronger evidence before the model classifies it as an elephant. Ré emphasizes the corresponding deployment assumption: the class proportions, class distributions, and estimated parameters in training must remain representative at test time. A classifier trained on a heavily skewed population can make systematically different decisions when deployed on a population with a different balance.

GDA is intentionally a small, constrained example of generative modeling. Its immediate value is that it makes a general trade-off legible. A model can tell a richer probabilistic story about where data come from, but restrictive structural assumptions can still reduce its eventual hard decision to a familiar linear classifier.

A Gaussian distribution has both a center and a shape. In one dimension, the mean $mu$ locates the center and the variance $sigm{a}^2$ determines how concentrated the density is:

A small variance produces a narrow, tall bump; a larger variance spreads the same total probability mass over a wider range. In multiple dimensions, the covariance matrix $Sigma$ takes over that role. Identity covariance produces circular contours. A diagonal covariance stretches contours along the feature axes. A full covariance can rotate them, with off-diagonal entries encoding correlation between features.

Ré adds an important qualification about covariance matrices: negative off-diagonal entries are compatible with a valid covariance matrix. The relevant requirement is positive eigenvalues, not that every entry be positive. The diagonal entries, as individual variances, must be non-negative.

GDA fixes the following generative story for binary labels:

The classes have different means, but share one covariance matrix. Their Gaussian contours therefore have the same shape, size, and orientation; they are centered at different points.

A student presses the obvious objection: real populations need not have the same variance or correlation structure. Ré agrees. Shared covariance is a modeling choice, not a literal claim that cats and elephants vary in the same way. It is also a consequential constraint. In a feature space of dimension $d$, covariance has on the order of $d^2$ entries, subject to symmetry. Estimating separate covariance matrices can therefore be expensive and unreliable in high dimensions.

The model predicts by choosing the label with the larger posterior:

The marginal $P(x)$ drops out because it is common to both candidate labels. At the decision boundary, the two remaining scores are equal:

Expanding the Gaussian log densities introduces quadratic terms of the form $x^{T}Sigm{a}^{-1}x$. Because both classes use the same $Sigma$, those terms cancel. What remains is affine in $x$:

The boundary is therefore a hyperplane: a line in the two-feature example. Shared covariance produces a straight separator because the quadratic terms cancel. If each class has its own covariance, the terms remain and can yield curved decision regions.

If each class instead has its own covariance, $Sigm{a}_{0}eSigm{a}_1$, the resulting model is quadratic discriminant analysis, or QDA. It is more flexible because it estimates covariance separately for each class, but it needs substantially more structure from the data. It can also produce decision regions in areas with no observations. Ré’s caution is not that QDA alone extrapolates: linear models make decisions far beyond their observed data too. Curved boundaries merely make the extrapolation look more conspicuous.

GDA models the joint distribution of features and labels:

The product over examples relies on the IID assumption. Taking logs turns the product into a sum. The notable convenience of this model is that maximum-likelihood estimates can be written directly rather than found through iterative optimization.

The class prior is just the fraction of class-1 examples:

The class means are the average feature vectors within their respective classes:

The shared covariance is the average within-class residual outer product:

For every example, the model subtracts the mean associated with that example’s label, then pools the residual structure across both classes. That pooling is the shared-covariance assumption in estimator form: the procedure does not preserve separate covariance estimates for cats and elephants.

Ré derives the class-1 mean by isolating the terms in the log likelihood that depend on $m{u}_1$. Setting the gradient to zero yields a sum involving $Sigm{a}^{-1}(x^{(i)}-m{u}_1)$. If $Sigm{a}^{-1}$ is invertible, the remaining sum must be zero, which gives the sample mean. The invertibility condition matters: if the covariance were singular, the vector could lie in a null space rather than itself being zero, and the parameter would not be identifiable by that argument.

A student asks whether this should count as learning when there is no gradient descent. Ré’s answer is yes. Learning, in this framework, is inference from data that supports future predictions. Gradient descent could optimize the same likelihood, but it would be a more expensive route to an answer that counting, averaging, and matrix operations already provide.

The closed form is a computational advantage, not a guarantee that the model transfers. GDA is cheap to fit because the Gaussian assumptions, shared covariance, and IID setup make its likelihood tractable. Its estimates are useful only to the degree that the data are sufficient to locate the class means and covariance, and that the learned population remains relevant at deployment.

The linear boundary is not the whole relationship between GDA and logistic regression. Under GDA’s Gaussian class-conditionals and shared covariance, the posterior itself has sigmoid form:

Here $heta$ and the intercept $b$ are determined by the class prior, the two means, and the shared covariance. Equivalently, the intercept can be absorbed by augmenting $x$ with a constant feature. The result is stronger than the observation that both methods can draw a line in the same place. Under GDA’s assumptions, logistic regression can express the same posterior probabilities.

The converse does not hold. Logistic regression does not require Gaussian class distributions or shared covariance. Its operative assumption is that the log-odds are linear in the features. Ré also notes that Gaussians are not unique in producing a logistic posterior: other exponential-family class-conditionals with shared structure can lead to the same form.

That asymmetry explains the classical appeal of logistic regression. It makes fewer claims about how each class generates observations, while still producing a useful linear probability model. The standard trade-off is that GDA’s stronger assumptions can make it more data-efficient when they are approximately correct, but more fragile when they are wrong. Logistic regression needs less of the data-generating process to be specified and is correspondingly more robust to a misspecified generative story.

Ré resists treating this as a reason to discard generative models. The modern importance of generative methods, in his account, comes from learning broad representations from large quantities of data, including unlabeled data, rather than merely locating a label boundary. Language models predict next tokens during pretraining; diffusion models learn to reverse an added-noise process. Those systems are not scaled-up GDA. The connection is conceptual: modeling how data are generated can produce capabilities that a narrowly discriminative classifier does not target.

The same generative framework applies to discrete features. For text classification, a document can be represented as a binary bag-of-words vector. Given a vocabulary of $d$ selected words, each coordinate records whether a word appears:

A vocabulary containing “buy,” “free,” “meeting,” “cs229,” and “homework” would encode “Buy free stuff!” by setting the first two coordinates to one. “CS229 homework” would set the final two coordinates to one.

The representation creates an immediate dimensionality problem. A binary vector of length $d$ has $2^d$ possible configurations. Ré gives typical dictionary sizes of 10,000 to 50,000 words. A full model of $P(xmidy)$ over every possible document is therefore not feasible.

Naive Bayes replaces the full joint distribution with conditional independence:

The true chain rule would condition each word on earlier words as well as on the class. Naive Bayes drops those dependencies. Given spam or non-spam, it treats the appearance of each vocabulary term as independent of the others.

Ré is explicit that this is almost certainly false for natural language. Negation illustrates the failure: “not the worst” cannot be understood by independently multiplying evidence from “not” and “worst.” The model does not represent their interaction. Yet the simplification transforms an exponential parameter problem into a linear one.

The model estimates a class prior and two word probabilities per vocabulary term:

There are $2d+1$ parameters rather than a separate probability for each of $2^d$ possible feature vectors. Maximum-likelihood estimation is again counting: count the fraction of spam emails, then, for each word, count the fraction of spam and non-spam emails containing it.

For a new email, the classifier compares class scores. In practice, it adds log probabilities rather than multiplying many small probabilities:

The log calculation exposes a practical failure of raw counts. If a word has never appeared in spam, its estimated spam probability is zero and its log probability is negative infinity. Laplace smoothing prevents that outcome by adding one to each binary outcome count:

The added two in the denominator reflects the feature’s two possible values, zero and one. Ré describes this as a simple form of regularization: it prevents zero probabilities and reduces confidence in sparse estimates. The resulting classifier is cheap to fit and evaluate not because it has learned the structure of language, but because it has deliberately declined to model that structure.