3Blue1Brown
3Blue1Brown is Grant Sanderson's video channel about math and adjacent fields such as physics and computer science, using animation to visualize core ideas and clarify difficult topics.
Cross-Entropy Measures the Cost of a Mismatched Probability Model
Grant Sanderson of 3Blue1Brown defines cross-entropy as the average coding cost of data generated by one probability distribution when encoded using another distribution’s assumed probabilities. In language-model pre-training, that cost is the average negative log probability assigned to observed next tokens, so minimizing it pushes the model toward the distribution underlying its data. In distillation, a smaller model minimizes cross-entropy against a larger model’s full probability distribution; KL divergence is the excess coding cost above the entropy of a matched code.
Shannon’s Entropy Limit Frames Language Models as Text Compressors
Grant Sanderson’s 3Blue1Brown video uses the question of how far English can be compressed to rebuild Shannon’s definitions of information and entropy. Sanderson argues that prediction and compression are mathematically equivalent: a good language predictor is, in principle, a good text compressor, and Shannon’s estimate of roughly one bit per English character frames the limit such systems are trying to approach. The result is a narrower version of the slogan “compression is intelligence”: not a definition of intelligence, but an explanation of why compression theory sits so close to modern language-model training.