Expected information gain: how to choose a guess

Transcript: this stretch, timestamped

The previous pages built the machinery — surprisal, entropy, cross-entropy, KL divergence — on toy distributions where the probabilities were handed to you. This page is where the machinery meets a problem that fights back. Nobody hands you the distribution in Wordle; you have to construct it out of the game's own rules, and the construction is the whole lesson. Everything downstream in this arc (priors from word frequencies, two-step search, and the bug that changed the punchline) is a refinement of the object built here: the pattern distribution induced by a guess.

Outline, with timestamps

• 00:00 — The game: five letters, six guesses, and a bot run that lands it in three.
• 02:31 — First instinct: pick openers that hit the most frequent letters. Unsatisfying, because it ignores position.
• 03:34 — Two word lists: ~13,000 legal guesses, ~2,300 curated answers, and a self-imposed rule not to use the second.
• 05:06 — One guess, many outcomes: a lucky pattern leaves 58 words, a boring one leaves 1,400.
• 06:37 — The full 3⁵-bar histogram, and the inversion: the likeliest patterns are the least informative.
• 08:10 — What a bit is, and the formula that falls out: information = log₂(1/p).
• 10:11 — Probability multiplies, information adds. That is the reason for the logarithm.
• 10:42 — Entropy as expected information: weary buys 4.9 bits, slate buys 5.8.
• 11:44 — Shannon, von Neumann, and how the quantity got its unhelpful name.
• 13:44 — Version 1 of the solver: maximise entropy every turn; simulate all 2,315 games; average 4.124.

The rules, and what a guess actually returns

Wordle hides a five-letter word and gives you six attempts. Each attempt must itself be a five-letter string the game recognises, and each attempt comes back painted: green in position i means your letter is the answer's letter in position i; yellow means the letter occurs in the answer but not there; grey means it does not occur at all (or, in the duplicate-letter case, does not occur again). Grant opens the video by letting his own bot play the 4 February 2022 puzzle, and it walks crane → stick → shard — 01:01.

The reframe that makes the rest of the video possible is to stop thinking of a guess as a probe of individual letters and start thinking of it as a single measurement with a five-symbol readout. You do not learn "is there an R"; you learn one string over the alphabet {grey, yellow, green}, all five positions at once, and the letters interact. That readout is the only channel between the hidden word and you, and everything below is a question about it: how many distinguishable things can it say, and how surprised should you expect to be by what it says?

Grant's first instinct is the one most people have — 02:31 — open with words that stack up the frequent letters of English, something like other then nails. It has an honest justification — even an all-grey response to a letter-dense guess is informative, because few words dodge all of those letters at once. But it cannot answer the question he immediately asks himself: is nails better or worse than snail (03:03)? Same letters, different arrangement, and letter-frequency reasoning is blind to the difference. (My addition: recomputing over the game's own guess list, uniformly weighted, nails is worth 5.65 bits and snail 5.45 — so his instinct that the trailing S is doing work was right, by about a fifth of a bit.) To settle questions like that you need a score, and the score is what the next fifteen minutes construct.

Two word lists, and the handicap Grant imposes

Wordle carries two vocabularies — 03:34. There is the list of strings it will accept as a guess, about 13,000 of them (12,953 in the word list checked into the video's repository), full of Scrabble-argument words nobody would choose as an answer. And there is a second, hand-curated list (04:05) of about 2,300 words — 2,315 in the video, 2,309 in the file the repository ships today — that the game will actually use as answers, in a fixed daily order.

That second list sits in the page source, in order, so you can read tomorrow's answer off it. Grant declines to use it — 04:35 — for two reasons worth separating. The weak one is sportsmanship. The strong one is generalisation: a solver tuned to one curated list is fitted to a test set and falls apart the moment the list changes hands, as it later did. So the plan is to use only universal information — which strings are legal words, and later how common they are in English — and keep the official answer list strictly as a benchmark.

A guess is a partition into 243 buckets

Fix a guess, say weary, and fix a set of candidate answers. Every candidate produces exactly one colour pattern against that guess. So the guess sorts the candidates into buckets, one bucket per pattern, and those buckets are a genuine partition: disjoint, exhaustive, and determined before you know the answer. The pattern you eventually see tells you exactly which bucket the truth is in, and nothing more. The guess is the partition. That is the whole model.

Under the working assumption that every candidate is equally likely, the probability of seeing pattern i is just the bucket's share of the candidates:

p(pattern i)  =  (number of candidates in bucket i) / (total candidates)

Grant makes this concrete with the two extremes of the weary distribution — 05:06. One lucky pattern (that unlikely W landing) leaves only 58 of the ~13,000 words standing, a probability of about 0.45%. A dull one — no W, no A, no R, no Y, an E somewhere — leaves 1,400, about 11%. And the single likeliest outcome, all grey, happens about 14% of the time. The inversion is the point:

"The point is the pattern with a lot of information is by its very nature unlikely to occur."— Grant Sanderson, 05:36

There are at most 3⁵ = 243 patterns, which is why the histogram at 06:37 has that many bars. But 243 is an upper bound on the alphabet, not a count of what can happen, and the gap matters in two ways. (Both numbers below are mine, computed against the repository's word lists.) First, five of the 243 patterns are structurally impossible for any pair of five-letter words: four greens plus a yellow. If four positions match exactly, every answer letter except one is already accounted for, so the fifth guessed letter either matches (five greens) or occurs nowhere unconsumed (grey) — it can never be yellow. That leaves 238 attainable patterns, and all 238 do occur somewhere in the real lists. Second, and far more restrictive, a single guess never comes close to 238: the most discriminating word in the list, trace, splits the 2,309 answers into just 150 non-empty buckets, and typical guesses manage fewer. The distribution is sparse and lopsided, and that lopsidedness is exactly what we are about to measure.

Bits: information = log₂(1/p)

Grant derives the unit rather than declaring it — 08:10. Start from the thing you actually care about: how many times did this observation halve my space of possibilities? Observing "the answer contains an S" is worth one halving, because roughly half the five-letter words have an S (45.7% of the guess list, in fact). Observing "contains a T" is worth two, because about a quarter do (23.4%) — 08:40. Write I for the number of halvings and p for the probability of the observation and the definition is forced:

(½)^I = p     ⟺     2^I = 1/p     ⟺     I = log₂(1/p) = −log₂(p)

All three forms are the same statement — 09:10. The −log₂ p form is the one you meet in textbooks and the one that looks arbitrary; the log₂(1/p) form is the one that reads as "how many halvings", and it is worth keeping that reading in your head, because it is what makes fractional bits sane. A pattern with probability 0.45% is worth log₂(1/0.0045) ≈ 7.8 bits: not a whole number of halvings, just a lot of them.

Why the logarithm at all, when the raw probability is right there? Grant gives one convenience reason and one structural reason — 09:41. The convenience is scale: "twenty bits" is easier to say and compare than 0.00000095. The structural reason is additivity, and it is the one that matters — 10:11. If your first guess cuts the space by 4 and your second cuts what remains by 8, the two together have cut by 32. Probabilities multiply; their logarithms add. So a quantity defined as a log turns a sequence of guesses — which is what a Wordle game is — into a sum you can budget against. That is why the currency is bits and not "words remaining", and it is the hinge of the whole design.

Entropy: the expected bits a guess buys

Now combine the two halves. The partition gives a probability for each pattern; the logarithm gives an information value for each pattern; take the expectation — 10:42:

H(guess)  =  Σ over patterns i   p(i) · log₂( 1 / p(i) )

This is Shannon entropy, applied not to a source of English text but to the readout of one guess. Read it as a scoring function: the bits I expect this guess to hand me, averaged over how the puzzle might go. It promises nothing about a single game — Grant's own run comes in under expectation twice running.

The comparison that makes it land is weary against slate — 11:13. weary's distribution is spiky: one fat all-grey bar at 14% and a long thin tail. slate's is flatter: its biggest bar is only about 6%, so even its worst outcome hands you log₂(1/0.067) ≈ 3.9 bits. Averaged out, weary is worth 4.9 bits and slate 5.8 — a difference of nearly a full halving of the search space, bought for free by typing different letters.

My computation, not the video's: entropy of the pattern distribution over the 12,953-word guess list taken as uniform, using the corrected pattern rules. It reproduces Grant's on-screen 4.9 for weary and 5.8 for slate, and the ≈14% and ≈6% grey bars he quotes. Note that snail and nails are anagrams: identical all-grey bar, different entropy, because rearranging the letters changes how the yellows and greens split the rest.

Two readings of the number itself — 12:44. Entropy measures flatness: for a fixed number of outcomes it is maximised by the uniform distribution, so a high-entropy guess is one whose outcomes are hard to predict. And it measures effective count: a distribution with entropy H is as uncertain as a fair choice among 2^H options — 16 equally likely patterns is 4 bits, 64 is 6 bits — 13:14. That gives the hard ceiling for a Wordle guess: uniform over all 243 patterns would be log₂(243) = 7.92 bits, and no guess can beat it. (Since only 238 patterns are achievable, the true ceiling is log₂(238) = 7.89 — and, as noted above, sparsity puts the real ceiling far lower.) Best openers land around 5.9, so a good guess is running at roughly three-quarters of a theoretical maximum that nothing can actually reach.

The name has a story — 11:44. Claude Shannon developed information theory at Bell Labs in the 1940s and, per the anecdote, had no good name for the expected-information quantity until John von Neumann suggested one:

"Your uncertainty function has been used in statistical mechanics under that name, so it already has a name. And in the second place, and more important, nobody knows what entropy really is. So, in a debate, you'll always have the advantage."— Grant Sanderson, 12:14

My addition, for honesty: this exchange is second-hand and undated, and there is no contemporaneous record of it. Treat it as folklore with a good moral rather than as history. The connection to thermodynamic entropy is real — both are Σ p log(1/p) over a distribution of microstates — but nothing on this page depends on it. For our purposes, entropy means one thing: the expected information value of a guess.

Why bits, and not "expected words eliminated"

There is an obvious competing score, and Grant explicitly flags it and sets it aside — 07:38. Instead of expected bits, minimise the expected number of candidates left standing:

E[remaining]  =  Σ over buckets i   p(i) · n(i)   =   Σ n(i)² / N     (uniform case)

It is cheap, it is intuitive, and it usually ranks guesses about the same way. But it is the wrong currency for three reasons.

It does not add. Counts multiply down a game tree; logs add. If you want to reason about "I need 11.2 bits and I have six guesses", you need a quantity whose per-guess values sum to the total, and only the log does that. Formally this is the chain rule for entropy: for a guess G and the hidden answer X, H(X) = I(X;G) + H(X | G) — the bits you gain now plus the bits you still owe. Maximising the first term is exactly minimising the second, and the identity telescopes across a whole sequence of guesses. Expected-count has no such decomposition.

The two criteria genuinely disagree. (Constructed example, mine.) Take 8 equally likely candidates. Guess A splits them 5·1·1·1; guess B splits them 4·2·2. Then A scores H = 1.549 bits with E[remaining] = 3.5, while B scores H = 1.500 bits with E[remaining] = 3.0. A wins on entropy, B wins on words eliminated. Neither is "correct" — they are different bets, one on the tail and one on the average — which is a useful early warning that entropy here is a heuristic for the score, not the score itself.

It does not survive a prior. Once words carry unequal probabilities of being the answer — the whole subject of the next page — "number of words remaining" stops being the quantity you care about, because a bucket holding forty obscure words can matter less than a bucket holding two common ones. Entropy takes the weights natively: you feed it probabilities, and it never needed the counts in the first place.

A worked example small enough to check by hand

Suppose play has narrowed the answer to eight equally likely words, all beginning sha: shard, sharp, shark, shawl, shame, shale, shape, share. You are owed log₂ 8 = 3 bits. Compare two guesses: shark, which is one of the candidates, and plied, which is not and cannot possibly be right.

Add up Σ p·log₂(1/p) in each block. For shark: 0.5(1) + 0.375(1.415) + 0.125(3) = 1.406 bits, from three buckets. For plied: 0.25(2) + 6 × 0.125(3) = 2.750 bits, from seven. The residual uncertainty confirms the chain rule exactly: after shark you still owe 3 − 1.406 = 1.594 bits, and indeed 0.5·log₂4 + 0.375·log₂3 + 0.125·log₂1 = 1.594; after plied you owe 3 − 2.750 = 0.250 bits, which is the single surviving two-word bucket weighted by its 1/4 chance.

Version 1: greedy, one step at a time

The algorithm is now a two-liner — 13:44. For every one of the ~13,000 legal guesses, bucket the current candidate set by pattern, compute the entropy of the resulting distribution, and play the argmax. After the real pattern comes back, throw away every candidate that does not match it and run the identical procedure on what is left — 14:14. The candidate set shrinks; the guess set never does.

The instrumented version at 14:45 shows two numbers side by side that are easy to confuse. On the right, expected information per candidate guess — the entropy defined above, computed before you play. On the left, the uncertainty remaining: the entropy over surviving words, which under a uniform prior is just log₂(number of candidates), and so 13.66 bits at the start because 2^13.66 ≈ 12,950 — 15:45. Both are in bits, which is the point of the whole design, so they subtract. His mid-game quiz makes the arithmetic explicit — 16:47: 5.78 bits of uncertainty (that is 2^5.78 ≈ 55 words) minus 4.7 bits gained leaves about 1 bit, and 1 bit of uncertainty means exactly two candidates and a coin flip.

Benchmark it by playing all 2,315 official answers — the test set the solver was never allowed to look at — and version 1 averages 4.124 guesses — 17:48. Par for a competent human is four. Not bad for a program that knows nothing about English except which strings are words.

Two flaws are already visible in the run, and they set up the rest of the arc. The solver has no endgame: down to two candidates it cannot use the fact that one is a common word and the other is not, so it grinds for information it does not need — the fix is a prior over words, which is P17. And the whole thing is greedy: it maximises the bits bought this turn, which is not the same as minimising the expected number of turns. A guess that buys slightly fewer bits now can leave a set that is much cheaper to finish, and only a two-step search finds those.

Where people get stuck

"243 patterns, so the entropy ceiling is 7.92 bits — why do good guesses only hit 5.9?" Because the ceiling assumes all 243 patterns are equally likely, and they are wildly not. Five are impossible for any word pair (four greens and a yellow), and beyond that, a single guess against the answer list produces at most 150 distinct patterns with a very lopsided distribution — all-grey alone eats 6–15% of the mass. 7.92 is the entropy of a distribution no Wordle guess can induce; it is a bound, not a target.

"Isn't it always better to guess a word that could be the answer?" No, and the worked table above is a counterexample: a guaranteed-wrong word bought 2.75 bits where a live candidate bought 1.41. Guessing a candidate mixes two goals — gathering information and winning immediately — and the immediate-win term is worth only p, the probability that particular word is right. When the candidate set is large, that term is negligible and pure information wins. When it is small (two or three words, with guesses to spare) it dominates. Greedy entropy maximisation ignores the win term entirely, which is one concrete way it falls short of optimal play.

"Fractional bits." 5.8 bits is not 5.8 halvings of anything you can point to; it is an expectation over outcomes that individually deliver 1.2 or 3.9 or 7.8 bits. The clean way to read a non-integer H is via effective count: 2^5.8 ≈ 56, so the guess is, on average, as good as narrowing to one of 56 equally likely buckets. That reading also rescues the "uncertainty" readout, where 2^H is literally the number of candidates when the prior is uniform.

"Duplicate letters." The colouring rule for a guess containing the same letter twice is genuinely fiddly: greens are assigned first and consume answer letters, then yellows are assigned left to right against whatever remains, and a repeated letter with nothing left to match comes back grey. Guess speed against answer abide and the first E is yellow, the second grey; against erase both are yellow. Getting this subtly wrong is exactly the bug that forced the follow-up video — it is a small enough edge case to survive a lot of testing and still shift a leaderboard.

Going deeper, verified

• Source code for the video — Grant Sanderson (2022) · simulations.py holds the pattern matrix, get_pattern_distributions and get_entropies; the two word lists are in data/. Everything on this page can be reproduced from it in about thirty lines.
• Solving Wordle using information theory — lesson page — 3Blue1Brown (2022) · the written companion, with the figures as static images.
• "Oh, wait, actually the best Wordle opener is not crane…" — 3Blue1Brown (2022) · the correction video: the duplicate-letter bug, and the two-step search that replaces the greedy heuristic. Covered on P18.
• A Mathematical Theory of Communication — Claude E. Shannon (1948) · the source. Section 6 derives H = −Σ p log p from three axioms; the "effective number of choices" reading used above is his.
• Optimal Wordle solutions — Jonathan Olson (2022) · exhaustive search rather than a greedy heuristic, with an interactive decision tree. The reference point for how much greedy entropy leaves on the table.

Exercises

1. Verify the toy table — By hand or in ten lines of code, reproduce the eight-candidate example: implement Wordle colouring (greens first and consuming, then yellows left to right), bucket the eight sha… words under shark and under plied, and confirm 1.406 and 2.750 bits. A good answer also checks the chain rule numerically: gained bits plus expected residual entropy should equal log₂ 8 = 3 in both cases, to the last decimal.
2. Rank your own opener — Download allowed_words.txt from the video's repository, take it as a uniform candidate set, and compute the entropy of whatever word you actually open with. Compare against the 4.87–5.87 band in the table above, and report your word's all-grey probability and its number of non-empty buckets. A good answer notices that entropy and bucket count are correlated but not the same ranking, and can name a pair where they disagree.
3. Find where greedy fails — Construct a candidate set and two guesses such that the higher-entropy guess leads to a worse expected number of total guesses. The 5·1·1·1 versus 4·2·2 split above is a starting point: work out how many further guesses each partition costs if you must name the answer exactly, and say which split you would rather have with only two turns left. A good answer states the objective it is optimising, since "fewest guesses" and "most bits" are different objectives.