"Compression is intelligence" — what the claim means

Transcript: this stretch, timestamped

This is the first three and a half minutes of a three-part series, and there is no maths in it yet. What it does instead is make a claim the rest of the series has to earn: that the cross-entropy loss used to pre-train large language models is not an arbitrary engineering choice but the same quantity Claude Shannon arrived at in 1948 when he asked how small a message could possibly get. Everything downstream — the robot on the moon, prefix codes, the definition of information, entropy, cross-entropy, KL divergence — is the argument for that claim. This page is the claim itself, plus the piece of machinery Grant promises and defers: the actual bridge between prediction and compression.

Outline, with timestamps

• 00:00 — The setup: ASCII spends 8 bits a character, cleverness gets you to about 4, and smarter schemes go lower still.
• 00:32 — But is there a floor? The question dates to Shannon in the 1940s, and the method matters more than the number.
• 01:03 — The hinge: LLM pre-training is next-token prediction under cross-entropy loss, and prediction and compression are mathematically equivalent.
• 01:33 — So pre-training can be reframed as building the best possible text compressor — which is where "compression is intelligence" comes from.
• 02:03 — The hedge: the slogan is hard to judge, the safer claim is weaker, and this is the first of a trilogy aimed at the noiseless coding theorem.
• 02:36 — Method: information and entropy will be discovered, not defined, because a definition handed over early spoils the story.
• 03:08 — The punchline to watch for: you cannot answer the compression question without engaging some notion of intelligence. Then, a warm-up.

The question: does text have a floor?

Start with the crude baseline. ASCII assigns every character a fixed-width code; in practice each one occupies a full byte, so plain English text costs 8 bits a character. That is obviously wasteful, because the characters are not equally likely — e and space dominate, z and q barely appear — and a code that gives common characters short strings and rare characters long ones does better. Grant's figure of roughly four bits per character is the right order for that kind of scheme: a Huffman code built on English letter frequencies alone lands a little above four bits per character.

My addition, since the video moves past it quickly: ASCII is properly a 7-bit code padded to 8 for byte alignment, so one of those eight bits is pure padding before you have done any information theory at all. And frequency is only the first move. Real compressors go further by exploiting context — u after q is nearly free, and the fourth occurrence of "entropy" in a paragraph is nearly free — which is exactly the door through which language models will eventually walk. gzip reaches roughly 2.5–3 bits per byte on English by pattern-matching alone, with no notion of what the text means.

So the sequence 8 → 4 → 3 → … invites the obvious question at 00:32: does this bottom out? A floor is a strange thing to hope for, because it would have to be a statement about English itself rather than about any particular algorithm, and it is not obvious at the outset that such a statement can even be phrased. Shannon's answer is that it can — and that the floor is a property of a probability distribution, not of a language.

Prediction and compression are the same object

Here is the load-bearing claim of the whole series, stated at 01:03 and then deliberately postponed:

"…one of the conclusions of information theory says that prediction and compression are mathematically equivalent. They turn out to be two sides of the same coin."— Grant Sanderson, 01:03

Grant says "later I'll explain exactly how that works," and Part 2 does. But the mechanism is worth having in hand now, because once you see it the rest of the series stops being a sequence of definitions and becomes a single argument. The bridge is arithmetic coding — discovered independently by Jorma Rissanen and Richard Pasco in 1976, and made practical by Witten, Neal and Cleary in 1987 — and it runs in both directions.

Direction one: model → compressor. A language model is a function that takes a context and returns a probability distribution over the next token. Nothing more. Call the message x₁ x₂ … xₙ. By the chain rule, the model assigns the whole message the probability

P(x₁…xₙ) = P(x₁) · P(x₂ | x₁) · P(x₃ | x₁x₂) · … · P(xₙ | x₁…xₙ₋₁)

and taking −log₂ of both sides turns that product into a sum:

−log₂ P(x₁…xₙ)  =  Σᵢ −log₂ P(xᵢ | x₁…xᵢ₋₁)

The right-hand side is, term for term, the cross-entropy loss the model is scored on during training. The left-hand side is a number of bits. That is the whole coincidence, and it is not a coincidence: the loss is a code length, measured in bits, of the training corpus under the model.

Arithmetic coding is the construction that collects those bits. Keep a live interval, starting at [0, 1). To encode the first token, chop the interval into pieces whose widths are the model's predicted probabilities, and keep the piece belonging to the token that actually occurred. Feed that token back into the model, get a new distribution, chop the surviving interval the same way, keep the right piece, and repeat. After n tokens the surviving interval has width exactly P(x₁…xₙ) — each step multiplied the width by one conditional probability. Then transmit the shortest binary fraction that lands inside it.

Pinning down a point inside an interval of width w takes about log₂(1/w) bits, and the standard bound — MacKay states it as the message length always landing within two bits of the information content of the entire source string — is that arithmetic coding never spends more than −log₂ P(x₁…xₙ) + 2 bits in total. Note where the "+2" sits: it is a constant for the whole message, not a penalty per symbol. That is precisely what a per-symbol code like Huffman cannot do, because a per-symbol code must round every code word up to a whole number of bits.

A small worked example, using a made-up model over a five-token message. The probabilities are the model's conditional predictions at each step:

The message probability in the bottom row is the product 0.20 · 0.05 · 0.02 · 0.30 · 0.60 = 3.6 × 10⁻⁵, and −log₂(3.6 × 10⁻⁵) = 14.762, which is exactly the column sum — the chain rule and the log doing their job. Arithmetic coding sends this in at most 17 bits; per-symbol rounding also costs 17, so on five tokens it is a tie. The point is what happens at length: on a 5,000-token document the rounding tax is up to 5,000 wasted bits while arithmetic coding still pays 2. Fractional bits only mean anything in aggregate, and arithmetic coding is the machine that lets you spend them.

Direction two: compressor → model. The converse is less often spelled out and is half of why the equivalence is interesting. Suppose someone hands you a black-box lossless compressor and lets you call it. Write L(s) for the length in bits of its output on string s. Then the compressor already implies a next-token distribution:

q(x | context)  ∝  2^−( L(context · x) − L(context) )

In words: the model's opinion about token x is read off from how many extra bits the compressor spends when you append x. A token the compressor finds cheap is a token it expected. The proportionality sign is doing real work — a general compressor's implied weights need not sum to one, though because it is uniquely decodable they can never sum to more than one, which is Kraft's inequality and shows up again in P2. Normalise over the vocabulary and you have a genuine probability distribution extracted from a program that was never told it was a model.

The loss number on your screen is a compression rate

This section is my addition; the video reaches it in Part 2. Training logs report cross-entropy loss in nats per token, because the implementation takes a natural log. Divide by ln 2 ≈ 0.6931 and you have bits per token, and bits per token is a file size. A model sitting at a loss of 2.0 nats/token is compressing its evaluation text to 2.0 / 0.6931 = 2.885 bits per token. If the tokeniser averages 4 bytes to the token, that is 2.885 / 4 = 0.72 bits per byte — against 8 bits per byte for raw storage and roughly 2.5–3 for gzip.

Do that arithmetic once by hand and the equivalence stops being philosophical: a loss curve going down is a file getting smaller, in the same units. It also explains why compression benchmarks are a live evaluation of language models rather than a curiosity — they are the training objective, reported without the log.

Why the slogan is slippery, and what survives

Grant states the strong version and immediately backs off it at 01:33, which is the correct move:

"The safer claim would be that the mathematical theory of compression is bizarrely relevant to artificial intelligence."— Grant Sanderson, 02:03

Three things go wrong with the strong reading, and naming them now keeps the rest of the series honest.

Compression is only defined relative to a decompressor. There is no such thing as "the size of a compressed file" on its own; there is the size of the file plus the program needed to unpack it. Otherwise the cheat is trivial — write a decompressor that already contains the corpus and ship a zero-byte archive. Any benchmark that wants compression to mean something must therefore measure the self-extracting archive: compressed data and decompressor together. The Hutter Prize is the standing example, and it is worth knowing precisely what it measures. Announced by Marcus Hutter in 2006 and expanded on 21 February 2020, it pays for losslessly compressing a fixed snapshot of English Wikipedia — originally enwik8, the first 100 MB, now enwik9, the first 10⁹ bytes. What is scored is the combined size S of the compressed archive and the decompressor that unpacks it, run under fixed time and memory limits. Against a standing record L, the award is 500,000 € × (1 − S/L), with a 1% improvement the minimum admissible claim. The eighth and current record, set by Kaido Orav and Byron Knoll in September 2024 with a decompressor called fx2-cmix, is 110,793,128 bytes — about 0.89 bits per byte of original Wikipedia. The prize's own stated rationale is the slogan under discussion: that being able to compress well is closely related to intelligence, and that a program which beats its predecessors on this file likely has to be smarter.

A file size is a statement about one corpus, not about generality. Lossless compression of a fixed snapshot rewards modelling the distribution you were handed, and says nothing directly about behaviour off that distribution, about planning, tool use, or calibration under shift. The implication runs one way and weakly: good compression requires good modelling, and good modelling is necessary but not sufficient for the things we call intelligence.

The interesting version of the claim is older than machine learning. Strip out the marketing and what remains is Occam's razor with units attached: the shortest program that outputs your data is your best explanation of it, and finding short programs is what understanding is. That is the Kolmogorov-complexity and minimum-description-length tradition; Legg and Hutter's "Universal Intelligence" (2007) is the most serious attempt to turn it into an actual definition, and Ilya Sutskever's 2023 Simons Institute talk "An Observation on Generalization" is the most-cited modern restatement. It is also why the claim resists falsification in its strong form — Kolmogorov complexity is uncomputable — and why the field settles for the safer version Grant states: not that compression is intelligence, but that the mathematics of compression keeps turning up, unbidden, in the mathematics of learning.

Definitions as the residue of insight

At 02:36 Grant explains the pedagogical shape of the next half hour, and it is worth flagging because it is unusual enough to be disorienting if you are not expecting it:

"…great definitions are often the residue of some kind of insight."— Grant Sanderson, 02:36

A normal treatment opens with H(p) = Σ p(x)·log₂(1/p(x)), verifies a few properties, and moves on. This one refuses to write the formula until minute 24. The bet is that if you chase the compression question honestly you will be forced into −log₂ p with nowhere else to go, and that arriving under duress is different from being told. If you already know the formulas, the payoff of these thirty minutes is not the formulas but the forcing — seeing which constraint kills every alternative.

The other thing to watch for is stated at 03:08: Shannon could not answer "how compressible is English" without engaging some notion of intelligence. That is not a flourish. The floor depends on the distribution you use, better distributions give lower floors, and the best available distribution over English is whatever the smartest available predictor believes. The floor is therefore not a fact about English but a fact about English and your model of it — which is why the question could not be finished in 1951 and is still moving.

Where the series goes from here

The arc is short and each step is forced by the previous one. As a map to keep beside you:

Grant calls this the first in a trilogy. Part 1 is this video, ending at the noiseless coding theorem — the standard textbook name is the source coding theorem; both refer to the same result — and a number for the entropy of English. Part 2 — "But what is cross-entropy?" — defines cross-entropy, applies it to the language-trees-and-zipping result, derives the pre-training objective, and closes on KL divergence. As of this writing the third has not been published, so this map covers Parts 1 and 2 and will be extended when the third lands.

Immediately next, though, the maths starts small. P2 puts a robot on a distant moon and gives it four possible instructions with probabilities ½, ¼, ⅛, ⅛. Everything above is going to fall out of that toy.

Where people get stuck

"Prediction and compression are equivalent" does not mean an LLM is a zip utility you can run today. The equivalence needs three things at once: a model, an entropy coder, and bit-identical determinism on both ends. The decoder has to reconstruct the encoder's intervals exactly, which means reproducing floating-point inference to the last bit across hardware, batch size, and kernel version. That is a genuine engineering obstacle, not a formality, and it is why LLM-based compressors are research demos rather than the thing on your laptop. The equivalence is mathematical; the deployment is hard.

Bits per token, bits per character, and bits per byte are three different numbers. LLM loss is per token; classical results like Shannon's are per character; compression benchmarks report per byte. Converting between them requires knowing the average tokens-per-byte of the tokeniser, and comparisons that skip this step are routinely off by a factor of three or four. Always ask which denominator a quoted rate is using.

Nats versus bits. A loss of "2.0" in a training log is almost always 2.0 nats, because the code called a natural log. In bits it is 2.0 / ln 2 = 2.885. Multiplying by 1/ln 2 ≈ 1.4427 is the conversion, and forgetting it makes a model look 44% better at compression than it is.

Smaller output does not by itself mean smarter. This is the decompressor problem again, and it is the single most common way the slogan is misused. A compressor that has memorised the evaluation corpus reports a spectacular ratio and has learned nothing. Every serious framing of the claim — the Hutter Prize's self-extracting archive rule included — exists to close exactly this hole.

Going deeper, verified

• Visual Information Theory — Christopher Olah (2015) · The post Grant credits in the description as the source of the visualisation of entropy used in this video; the best-drawn introduction to codes, entropy and KL that exists.
• A Mathematical Theory of Communication — Claude Shannon, Bell System Technical Journal (1948) · The founding paper. Sections 6–9 contain the definition of entropy and the noiseless coding theorem this video reconstructs from scratch.
• Prediction and Entropy of Printed English — Claude Shannon, Bell System Technical Journal vol. 30 (1951) · The measurement, made with human guessers standing in for a language model. With 100 letters of context Shannon's experiments bracket English between about 0.6 and 1.3 bits per letter. Prediction used as a measuring instrument for compressibility is the direct ancestor of everything on this page.
• The Hutter Prize for Lossless Compression of Human Knowledge — Marcus Hutter (2006, expanded 2020) · The slogan operationalised: compress the first gigabyte of English Wikipedia, scored on the combined size of archive plus decompressor. Read the rules page specifically for how it defends against hiding data in the decompressor. Note the site's HTTPS certificate is expired — the plain http:// link above works.
• Information Theory, Inference, and Learning Algorithms — David MacKay, Cambridge University Press (2003), free PDF · Chapter 5 is Huffman and its rounding losses; §6.2 is arithmetic coding, including the two-bits-for-the-whole-message bound quoted above. The single best follow-on if this page's middle section is the part you want to go deeper on.

Exercises

1. Turn a loss into a file size — Take a model reported at 2.4 nats/token on a corpus whose tokeniser averages 3.8 bytes per token. Compute its bits per token and bits per byte, and compare against 8 bits/byte for raw storage and about 2.8 for gzip. A good answer states the conversion factor 1/ln 2 explicitly, gets 3.463 bits/token and 0.911 bits/byte, and says in one sentence why that number is a claim about the corpus as much as about the model.
2. Build the interval by hand — Using the five-token table above, compute the arithmetic coder's surviving interval step by step, starting from [0, 1) and assuming the encoded token occupies the bottom slice at each step. Verify the final width equals 3.6 × 10⁻⁵, then find the shortest binary fraction inside it and count its bits. A good answer confirms the count is at most ⌈14.762⌉ + 1 = 16 bits and explains why the bound is a constant rather than per-symbol.
3. Run the cheat — Compress a text file with gzip and record the size. Now concatenate the file with a second copy of itself and compress that. The doubled file should compress to only slightly more than the original. Explain the result in terms of −log₂ P: what probability did the compressor assign to the second half, and what does that say about the difference between "the size of the data" and "the size of the data given a model that has already seen it"? A good answer connects this directly to why the Hutter Prize counts the decompressor.