Grant Sanderson
Math educator and creator of 3Blue1Brown, a visual-mathematics YouTube channel. He creates and narrates lessons on mathematical foundations relevant to AI, including information theory, neural networks, and cross-entropy.
Cross-Entropy Measures the Cost of a Mismatched Probability Model
Grant Sanderson of 3Blue1Brown defines cross-entropy as the average coding cost of data generated by one probability distribution when encoded using another distribution’s assumed probabilities. In language-model pre-training, that cost is the average negative log probability assigned to observed next tokens, so minimizing it pushes the model toward the distribution underlying its data. In distillation, a smaller model minimizes cross-entropy against a larger model’s full probability distribution; KL divergence is the excess coding cost above the entropy of a matched code.
AI Math Progress Is Jagged, Not a Clean AGI Benchmark
Grant Sanderson argues that AI’s rapid gains in mathematics are less a clean proxy for AGI than a map of uneven capabilities: solving contest problems, finding cross-field connections, inventing definitions, verifying proofs, and explaining ideas are different tasks with different signals. In conversation with Dwarkesh Patel, Sanderson says the most consequential mathematical breakthroughs may be the hardest for current systems to learn, because their value often depends on delayed judgment, human taste, and concepts that compress a field rather than merely prove a theorem.
Shannon’s Entropy Limit Frames Language Models as Text Compressors
Grant Sanderson’s 3Blue1Brown video uses the question of how far English can be compressed to rebuild Shannon’s definitions of information and entropy. Sanderson argues that prediction and compression are mathematically equivalent: a good language predictor is, in principle, a good text compressor, and Shannon’s estimate of roughly one bit per English character frames the limit such systems are trying to approach. The result is a narrower version of the slogan “compression is intelligence”: not a definition of intelligence, but an explanation of why compression theory sits so close to modern language-model training.