Orply.

Pretraining Minimizes the Bits Needed to Encode Text

Grant Sanderson3Blue1BrownTuesday, November 17, 202612 min read

Arithmetic coding turns a language model’s predictions into a bit cost: text the model finds more predictable takes fewer bits to encode. In his article, Grant Sanderson argues that this link makes pretraining legible as an effort to compress text and offers a way to examine aspects of intelligence. He stops short of equating the two, presenting compression as a mathematical lens on intelligence rather than its full definition.

Context, not character frequency, makes extreme compression possible

A fixed code assigns the same bit string to a character wherever it appears. It can shorten text by giving common letters shorter codes, but it cannot account for the fact that a word’s predictability depends on what came before it. In the preamble to the U.S. Constitution, for example, once “We the People of the United States” establishes the passage, much of what follows is highly predictable. The character-by-character Huffman code in Grant Sanderson’s demonstration still spends about 1,500 bits on the passage; a context-sensitive method can represent it in a few dozen.

The difference is not a cleverer fixed code for each letter. The method uses a predictor—a language model that assigns probabilities to what might come next—and turns those probabilities into a code. The more accurately the model anticipates the text, the smaller the encoded message can be.

That gap helps explain why Shannon’s information formula can sound implausible when applied to a recognizable passage. A message assigned probability one-half has one bit of information; a message assigned probability one in a thousand has about ten bits. If the remaining words of a famous document seem predictable, the formula appears to say that an entire paragraph might take only a handful of bits. Yet ordinary encodings use thousands.

The apparent contradiction comes from asking a fixed code to do a job it was not designed for. Huffman coding can exploit overall character frequencies, but it does not change its code in response to the surrounding text. A model can. After a particular context, it may assign nearly all the probability to one continuation. If that continuation is what appears, it costs little to encode.

For clarity, the demonstration first uses a small language model that predicts one character at a time, with only the 26 lowercase letters in its vocabulary. It leaves out capitalization and punctuation. The same method works with larger token vocabularies, which the later examples use.

A text becomes a narrowing interval

Imagine laying the model’s probabilities for the next character across the interval from zero to one. Each possible character gets a slice whose width equals its probability. A likely character occupies a broad slice; an unlikely one gets a narrow one.

After the first character is chosen, the model predicts the next character given that context. Its new probability slices are drawn inside the slice for the first character. The process repeats: each additional character selects a smaller subinterval. The width of the final region is the model’s probability for the whole string, because the probabilities of successive choices multiply.

For “math,” the first choice selects the region for m. Given m, the model’s next-character distribution is dominated by vowels; it assigns a a 38% chance and e 16%. The next distribution is conditioned on ma, then on mat. Each step narrows the interval to the region corresponding to the text seen so far.

At the end, any number inside the interval for “math” identifies that word, provided the receiver knows how many characters to decode. The sender can transmit a number in the interval along with the message length. The receiver starts from the same initial distribution, finds which character contains the number, feeds that character back into the model, and repeats until it has recovered the stated number of characters. An end-of-message symbol could replace the separate length field, but Sanderson keeps the length explicit for the examples.

The sender and receiver must use exactly the same predictor. If their models produce different probabilities at any step, they will divide the interval differently, and the number may decode to different text.

A uniformly chosen number in this interval picture tends to decode into plausible text rather than arbitrary characters. Likely strings occupy more of the number line, so a random point is more likely to land in a likely string’s region. Sanderson presents this as a way to visualize how language models generate text: they repeatedly choose from probability distributions, with more probable continuations occupying more space.

To turn the number into a practical message, the sender writes it in binary. Each additional bit halves the remaining range of possible values: the first bit chooses the left or right half of the unit interval, the next chooses a half of that half, and so on. A finite bit string therefore identifies a range of possible numbers. The encoding works when that range falls inside the target interval for the text.

For “math,” the demonstration finds a 10-bit value inside the correct region, or 2.5 bits per character. More strikingly, the same 10-bit value also falls inside the interval for “mathematics.” The added letters are so predictable that they barely shrink the interval. Ten bits for ten letters is about 0.91 bits per character, though the message length still has to be conveyed.

This is unusually fortunate. The interval for “mathematics” is about 2^(-18) of the number line, while values expressible with 10 bits are spaced about 2^(-10) apart. The word would ordinarily need around 18 bits of precision, but one of the 10-bit values happens to land inside its interval.

The probability calculation gives a sufficient bound, not a fixed code length

Values expressible with k bits or fewer are spaced 2^(-k) apart. If a target region has width p, choosing k so that 2^(-k) is smaller than p gives a sufficient worst-case bit count: a binary value can be found inside the region regardless of where it falls on the number line. Rearranging gives k ≥ −log₂(p). Rounding up the negative base-two logarithm of the model’s probability therefore guarantees enough bits to represent the text. Sometimes fewer bits will work by chance, as in the “mathematics” example.

That expression resembles Shannon’s information measure. It explains why arithmetic coding can come close to the limit in a concrete encoding scheme: the precision required to identify a point in a region is tied to the negative logarithm of the region’s width. For long messages, the separate length field is small relative to the encoded number. Sanderson gives a 32-bit length header as one example, sufficient for messages up to about four billion characters.

But “Shannon’s limit” needs qualification. The probability in this calculation is the probability assigned by the language model, not necessarily the true probability of the text under the distribution of messages a sender might produce. The compression rate therefore depends on how well the model captures language. A poor predictor gives a text a low probability and makes it expensive to encode; a good predictor makes the same text cheaper.

The occasional encodings shorter than −log₂(p) do not contradict Shannon’s theorem. The theorem concerns average code length under the relevant true distribution, not a requirement that every individual message use at least its information value in bits. Some strings land in a favorable position relative to the available binary values; others require rounding up to a longer code. Those cases balance out across messages.

The model’s predictions explain why this method is useful for text. A context-sensitive model can distinguish a likely continuation from a surprising one even when their individual characters have similar overall frequencies. In Sanderson’s Qwen demonstration, a 494-million-parameter model encodes a 297-character version of the Constitution preamble in 39 bits. This is a result for that model and passage, not a general benchmark. The first phrase, “We the People of the United States,” accounts for most of the decisions. After that, many tokens dominate the probability space, so the encoded number changes very little. Even the spelling choice between “defence” and “defense” appears as a small distinction in the model’s distribution.

Arithmetic coding turns the interval idea into an algorithm

The interval construction is conceptually direct, but following it literally would require ever more precise real numbers and increasingly precise boundaries. Grant Sanderson traces the idea at least to unpublished work by Peter Elias around 1960. In 1976, Jorma Rissanen and Richard Pasco independently developed a practical version that avoids the need for unbounded numerical precision: arithmetic coding.

Its central move is to renormalize the interval at each step, rescaling the current region so it again fills a unit-length interval. That keeps the computation manageable while preserving the essential probability-based narrowing process. The result is a way to turn a predictor into a compressor: plug in a model, and its successive predictions determine the coding intervals.

Sanderson says this is the structure behind the Hutter Prize submissions he mentions, where a predictor is paired with an arithmetic coder. The prize measures compression of a gigabyte of Wikipedia and includes the size of the decoding program. In the displayed results, V. Ivanov’s submission reaches 100 MB and earns €37,300. The decoder matters: without counting its size, a “decoder” could simply contain a hard copy of Wikipedia and return it regardless of the encoded bitstream.

He also notes a distinct approach, asymmetric numeral systems, published by Jarek Duda in 2013. Rather than representing the message as a point in a real interval, it encodes it in a large integer that grows as the message is processed. Sanderson describes it as avoiding the real-number precision issue and allowing a faster encoding process.

For Sanderson, arithmetic coding is more than a compression technique. It gives a concrete way to see how a model’s predictions translate into bits—and, in the other direction, how compression can imply predictions.

Pretraining can be read as learning to compress

The total code length for a text is approximately the negative logarithm of its probability under the model. Since the text probability is the product of the model’s successive conditional predictions, taking the logarithm turns that product into a sum: one cost for each next-token prediction. A prediction that assigns high probability to the actual next token has a low cost; assigning it low probability has a high cost.

The average of these costs over training tokens is cross-entropy. In the usual framing, pretraining aims to build a next-token predictor, and cross-entropy is the loss used to measure how well it predicts. Grant Sanderson offers a mathematically equivalent but different way to think about the objective: the model is being trained to make text compressible. Under that view, minimizing cross-entropy is minimizing the number of bits needed to encode the training examples.

Sanderson’s point is that the objective can be described as creating the best possible text compressor, rather than prediction for its own sake. In this framing, minimizing cross-entropy is not merely one possible choice of loss function; it corresponds to reducing the number of bits needed to encode the examples.

The model comparisons illustrate how much prediction quality matters. A 3.7-million-parameter NanoGPT model encodes the sentence “The concept of proof and its associated mathematical rigor began in ancient Greek mathematics, exemplified in Euclid’s Elements” in 254 bits, about two bits per character. Qwen, with 494 million parameters, encodes the same 127-character text in 96 bits, about 0.75 bits per character. In Sanderson’s example, the final words are especially cheap under Qwen because the context makes them predictable.

This raises an obvious concern: a large model might compress a familiar passage because it has effectively memorized it. Sanderson agrees that this is a real concern, and points to the Hutter Prize’s accounting as the relevant constraint. The score includes the size of the decoding program required to recover the data. If a hard-coded copy of Wikipedia were placed in that program, its size would count; a tiny payload would not make the full package small. Sanderson uses this accounting to explain how a model that has memorized a passage would be part of the cost of decoding it. He does not claim that model size alone demonstrates how much a model has memorized.

The point is not that every language model is a useful compressor, or that compression alone settles what intelligence means. It is that the coding scheme makes the relationship between prediction quality and compressed size explicit. A model’s predictions become a measurable bit cost.

Compression and intelligence overlap, but are not identical

Grant Sanderson treats “compression is intelligence” as a proposed mathematical definition, not a claim that the everyday meanings of the words are interchangeable. He compares the move to giving “energy” a precise meaning in physics: the definition is narrower than ordinary usage, but may still unify phenomena and prove productive. A mathematical account of intelligence, he suggests, might likewise omit human nuance while clarifying something important.

His first connection is between prediction and compression. Arithmetic coding turns a predictor into a compressor. Sanderson also sketches the reverse: take a compressor and compare the compressed size of a passage with and without a possible continuation. If adding that continuation increases the compressed message by Δ bits, then 2^(-Δ) approximates the compressor’s implied probability for it. Across possible continuations, those values can act like a predictive distribution. In this sense, prediction and compression can be translated into one another.

That connection also echoes predictive processing, a theory Sanderson describes as the brain generating top-down predictions about sensory input and receiving back the error between prediction and observation, rather than the full signal. He suggests that some of what appears to be intelligence may involve this combination of prediction and compression.

His second connection is that understanding often means reducing many facts to a small set of principles. In physics, shorter laws can feel like progress even when they make the same predictions. In machine learning, if two models predict equally well but one requires far more parameters, the smaller one may be relying on fewer memorized assumptions. Sanderson presents that as an interpretation of what model compactness might indicate, not as a result established by the compression comparison itself.

The analogy has a practical test. If two models have equal predictive ability, arithmetic coding should encode a given dataset to roughly the same size, because the coding rate depends on the predictions. But a fair comparison must also account for the decoding program needed to recover the data. A compact model with the same predictive power may then have an advantage. Sanderson connects this possibility to the idea that a clearer understanding uses fewer general principles instead of storing a large number of separate facts.

His third connection is the ability to learn quickly. A static compression score does not directly measure how long it took a model to train. But the score can change if the model adapts during encoding. A model trained on Wikipedia may predict Shakespeare less well than one tailored to Shakespearean English. If the encoder updates the model as it reads the new text, it can become better suited to that context. The decoder must replay exactly the same deterministic updates so its predictions remain synchronized. Sanderson notes that both sides can even begin from an identical pseudorandomly generated model, avoiding the need to transmit its initial parameters.

Under this setup, compression depends not just on the model’s starting predictive capacity, but on how quickly it adapts to the new material. Faster learning can produce better compression. Sanderson presents this as a third way compression may capture something associated with intelligence, alongside prediction and the ability to represent many facts with fewer general principles.

The conclusion is deliberately limited. Sanderson’s account of the trilogy is that studying compression repeatedly leads back to questions about intelligence: estimating the compressibility of English requires human prediction; modern AI training uses mathematical tools developed in the study of compression; and powerful compression methods depend on models that predict well, stay compact, and learn quickly. Still, he rejects literal equivalence. Compression is not the same thing as intelligence, just as a mathematical definition of energy is not the full everyday meaning of the word. What surprised him is how many aspects of intelligence the mathematical idea of compression can illuminate.

The safe answer is of course no, just as Newton's definition of energy is not the same as Webster's.

Grant Sanderson · Source

The frontier, in your inbox tomorrow at 08:00.

Sign up free. Pick the industry Briefs you want. Tomorrow morning, they land. No credit card.

Sign up free