Entropy, mapped

Six ideas, in the order the series earns them. If you read nothing else, read this list — everything below is one of these six unpacked at length.

• 1 · SurpriseA message carries information in proportion to how unlikely it was. Something you already expected tells you nothing. The amount of surprise in an outcome of probability p turns out to be forced to be log₂(1/p) bits — not chosen, forced, once you insist that independent surprises add. P4
• 2 · CodesA short code word is a bet that its symbol is common. Prefix-free codes cost you a budget that must sum to 1, so making one word shorter makes another longer. The best you can do is spend log₂(1/p) bits on a symbol of probability p. P2 · P3
• 3 · EntropyEntropy is the average surprise: H(p) = Σ p(x)·log₂(1/p(x)). It is the floor on bits per symbol that no encoder can beat, and the yardstick against which every compressor is judged. P6
• 4 · Cross-entropyCross-entropy is what you actually pay when your model of the world is wrong. Encode with a code built for q while reality is p and you pay H(p,q) = Σ p(x)·log₂(1/q(x)) — never less than H(p), and equal only when q = p. P9 · P10
• 5 · KL divergenceThe gap between them is the fine you pay for your wrong beliefs: D(p‖q) = H(p,q) − H(p). Zero if and only if you were right, asymmetric, and not a distance. P15
• 6 · Where the loss comes fromTraining a language model minimises cross-entropy, which is the same thing as building the best possible text compressor. That is not an analogy — it is the same number, and it is why the objective is the log of the probability of the right token and nothing else. P12 · P13

Each page sets its own exercises. These are the ones worth doing even if you only skim.

1. Build the robot's encoder. Take the ½, ¼, ⅛, ⅛ distribution from P2, implement the prefix code, encode ten thousand samples and measure the bits per symbol. Confirm you get 1.75 and not 2. Then change the distribution to be uniform and watch the advantage vanish.
2. Measure the entropy of English yourself. Take any book from Project Gutenberg. Compute the entropy of the letter distribution, then of letter pairs given the previous letter, then of triples. Watch the bits-per-character number fall with each order, and compare against the estimates in P5.
3. Show the compressor and the model are the same object. Take any small language model, compute its average cross-entropy in bits per character on a held-out text, multiply by the character count, and compare against what gzip, xz and zstd produce on the same file. Details in P12.
4. Play the Wordle solver. Implement the entropy-of-the-split scoring from P16 over the official word list, look at your top openers, then re-read P18 and check your pattern-matching code against the duplicate-letter cases that bit Grant.
5. Convince yourself cross-entropy is a proper scoring rule. Fix a true distribution over three outcomes, then grid-search over reported distributions and plot the expected loss. Confirm the minimum sits exactly at the truth, and that squared error does not have the same property in the same way. Argument in P13.