Defining cross-entropy

Transcript: this stretch, timestamped

P08 closed the loop on optimal codes: given a distribution, the best achievable cost per symbol is its entropy, and the way you achieve it is to spend log₂(1/p(x)) bits on symbol x. That is a statement about a matched pair — one distribution, one code built for it. Every interesting engineering situation is a mismatched pair: you compress with a model of English and get handed Portuguese, you train a network on one distribution and deploy it on another, you gzip a file with a dictionary learned from the first half. This page takes that mismatch and turns it into a number. It is the definitional page of the arc, so it is deliberately careful rather than intuitive — P10 does the intuition, the 90/10 examples, and the asymmetry.

Outline, with timestamps

• 05:20 — The plan changes: the space agency rotates the robot's walk, so up and down become 1-in-8, left 1-in-4, right one-half — but the encoder is hard-coded.
• 05:41 — Counting the damage: the old codeword lengths, re-weighted by the new frequencies, come to 2.625 bits per instruction.
• 06:13 — Naming it: that number is the cross-entropy — "you optimised for one context; how does it do in another?"
• 06:44 — Letters assigned: p for the new distribution, q for the one the code was built from; generalise past four symbols to any alphabet.
• 07:15 — The matched case as a reference: weighting q's code lengths by q's own frequencies is just the entropy H(q) from part one.
• 07:47 — The definition, and a warning: swap the weights for p and you have cross-entropy by definition; the notation in the wild is a mess.
• 08:19 — The picture: bars of width p(x) and height log₂(1/q(x)); the total area is the cross-entropy.

The setup: the plan changed, the codebook didn't

Part one's robot took a biased random walk: up half the time, down a quarter, left and right an eighth each. Those probabilities produced a beautifully clean optimal code — one bit for up, two for down, three each for left and right — because every probability was already a power of one half. Call that distribution q. It is the distribution the codebook was built from, and from here on that is the only fact about it that matters.

Now the mission changes (05:20). The agency wants a more rightward journey, so the walk is effectively rotated: right becomes the half-probability symbol, left the quarter, and up and down drop to an eighth each. Call this new distribution p. Crucially, the transmitter and receiver are hard-coded — nobody is going to fly out and reprogram them — so the old codebook keeps running against the new traffic.

Notice how narrow the question now is. Nothing about the code has changed; nothing about the decoder has broken; the messages still arrive unambiguously. The only thing that changed is how often each codeword gets used. So the cost per symbol is a weighted average with the same list of lengths and a new list of weights. That sentence is the whole definition, and everything below is bookkeeping on it.

Counting the damage, symbol by symbol

The arithmetic is small enough to do by hand, and doing it by hand is the fastest route to never mixing up the two slots again (05:41). The code length of a symbol is set by q: ℓ(x) = log₂(1/q(x)). The number of times you pay it is set by p.

Two more numbers make the table mean something. The old code on the old traffic cost H(q) = ½·1 + ¼·2 + ⅛·3 + ⅛·3 = 1.75 bits — that was part one's answer. And the best possible code for the new traffic would cost H(p) = ⅛·3 + ⅛·3 + ¼·2 + ½·1 = 1.75 bits as well, because p is just q with the labels shuffled and entropy does not care which symbol carries which probability. So the mismatch costs exactly 2.625 − 1.75 = 0.875 bits per instruction — a 50% inflation of the bill, paid for nothing but pointing the codebook at the wrong distribution. That excess is the Kullback–Leibler divergence D(p‖q), which the arc gets to properly later; here it is enough to see that it is a difference of two things this page has already computed.

The definition, with both slots pinned down

Generalise past four symbols (06:44). Let the alphabet be any finite set of symbols — robot moves, English characters, GPT tokens — and let p and q be two distributions over it. Then:

   H(p, q)  =   Σ   p(x) · log₂( 1 / q(x) )   =   −  Σ   p(x) · log₂ q(x)
                x                                     x

            =   𝔼         [ log₂( 1 / q(x) ) ]
                 x ∼ p

The expectation form on the second line is the one to memorise, because it names the two roles out loud: the thing being averaged is a property of q, and the distribution you average under is p. Written as a slot diagram:

   q  →  the MODEL     →  fixes the PRICES     ℓ(x) = log₂(1/q(x))   [inside the log]
   p  →  REALITY       →  fixes the RATES      how often ℓ(x) is paid [outside the log]

Say it several ways until one of them sticks. You pay q's prices at p's frequencies. You encode with q, sample from p. q is your belief; p is the world. The model is the one you can change, and it is the one buried inside the logarithm — which is exactly why, when this becomes a training loss in P13, the gradient flows through the q slot and the p slot is just data.

The matched case is the sanity check (07:15). Put the same distribution in both slots and the code was built for exactly the traffic it sees: H(p, p) = Σ p(x)·log₂(1/p(x)) = H(p). Cross-entropy is a two-argument generalisation of entropy, and entropy is its diagonal. My addition, not Grant's: the diagonal is also the floor — H(p, q) ≥ H(p) for every q, with equality only when q = p. That is Gibbs' inequality, and it is the formal reason "the true distribution is the best possible codebook" is a theorem and not a slogan. P08's coding argument and this inequality are two views of the same fact.

The picture: same bars, one axis swapped

Part one drew entropy as an area (07:15): one bar per symbol, width q(x), height log₂(1/q(x)), so each bar's area is that symbol's contribution to the weighted sum and the total area is H(q). Because width and height are locked to the same distribution, the bars have a characteristic shape — the wide ones are short and the tall ones are thin.

Cross-entropy is the same diagram with the widths taken from a different distribution (08:19): height stays log₂(1/q(x)), width becomes p(x). Now nothing forces tall bars to be thin, and total area can only go up. In our table, the two three-bit bars — the ones the old code treated as rare — got widened to cover 75% of the traffic, and that single fact is the whole 0.875-bit penalty. This is the picture worth carrying: cross-entropy is entropy's area diagram with the heights and the widths sourced from different distributions, and the excess area over the best-possible packing is the divergence.

The notational trap, and what libraries actually do

Grant flags this and then declines to fight it, which is the right call for a video (07:47):

"The specific notation that you'll see out in the wild is a little bit of a mess, there's a lot of different conventions."— Grant Sanderson, 07:47

Here is the concrete version of that warning. In mathematical writing, H(p, q) almost universally means true distribution first, model second — the formula at the top of this page. The arguments are not interchangeable: H(p, q) ≠ H(q, p) in general, so writing them the wrong way round is not a stylistic slip, it computes a different number. But the prose orderings drift even when the symbols do not. Grant's own sentence at 07:47 is "the cross-entropy of the distribution q relative to the distribution p", naming the code's distribution first, while much of the machine-learning literature says "the cross-entropy of p relative to q" for the same quantity. Do not try to arbitrate this. Anchor on the roles — which distribution set the code lengths, which one sets the frequencies — and re-derive the word order every time.

The libraries are worse, because there the order is executable. Two verified examples, opposite to each other:

• PyTorch puts the model first. torch.nn.functional.cross_entropy(input, target) documents input as "predicted unnormalized logits" and target as "ground truth class indices or class probabilities" — so the call is (q, p), the reverse of H(p, q). Same for kl_div(input, target), where input is the model's log-probabilities.
• SciPy and scikit-learn put reality first. scipy.stats.entropy(pk, qk) computes Σ pk·log(pk/qk), matching D(p‖q); sklearn.metrics.log_loss(y_true, y_proba) takes the labels first.

One more practitioner detail that bites: units. Everything on this page is in bits because the logs are base 2. Every one of those library functions uses the natural logarithm, so it returns nats. A language-model loss of 2.1 is 2.1 nats ≈ 3.03 bits per token; divide by ln 2 ≈ 0.693 to convert. Base changes scale cross-entropy, entropy and KL by the same constant, so no comparison between them is affected — but a number quoted with no unit is not yet a number.

Two edges the definition forces on you

A model that rules out something that happens pays an unbounded price. If q(x) = 0 for some symbol with p(x) > 0, then log₂(1/q(x)) = ∞ and the whole sum is infinite. The coding reading is exact and not a technicality: a code built from q reserves no codeword at all for x, so when x shows up you cannot transmit the message at any finite length. This is why every implementation keeps q strictly positive. PyTorch does it structurally — it takes logits and applies softmax internally, and a softmax output is never exactly zero unless a logit is −∞. scikit-learn does it by clipping y_proba into [eps, 1−eps] at machine precision. Label smoothing (PyTorch's label_smoothing argument) is the same defence applied to the other slot. If you ever see a loss of inf or nan on the first step, this is usually the reason.

The reverse edge is free. If p(x) = 0, the term is 0·log₂(1/q(x)), which is taken to be 0 by the standard convention (justified by the limit t·log(1/t) → 0 as t → 0⁺) — even when q(x) is tiny and the log is huge. So a model may believe in events that never occur and pay no direct penalty for them. It still pays indirectly: probability mass spent on the impossible is mass unavailable to the actual symbols, which lengthens their codewords. The asymmetry between these two edges is the seed of the "mode-covering vs mode-seeking" distinction you meet later with KL.

Where people get stuck

"Which one is the model again?" — the confusion this page exists to kill. Two independent checks that never fail. (1) Only q appears inside a logarithm; code lengths are logs of the model's probabilities, so whatever is inside the log is the codebook. (2) Weights in an average must be the frequencies things actually happen at, and only reality can supply those, so the bare multiplier out front is p. If a formula has the model outside and reality inside, it is H(q, p) — a legitimate quantity, but a different one.

"This example gave the same answer both ways, so cross-entropy must be symmetric." — a real trap set by the toy numbers. Compute H(q, p) from the table: ½·3 + ¼·3 + ⅛·2 + ⅛·1 = 2.625, identical to H(p, q). That is an accident of this example: p is obtained from q by swapping up↔right and down↔left, and that relabelling is its own inverse, which forces the two numbers to agree. Change one probability and the coincidence dies. Grant demolishes the symmetry assumption with a 90/10 example immediately after this stretch — that is P10's job.

"Is log₂(1/q) the same as −log₂ q, and why write it the awkward way?" — algebraically identical; rhetorically not. Since q(x) ≤ 1, log₂(1/q(x)) ≥ 0, so the reciprocal form makes every term a manifestly non-negative length in bits, which is what it physically is. The −log form saves ink and hides a sign you then have to remember. Both appear constantly; read them as the same thing.

"2.625 bits is not a whole number of bits — what is being transmitted?" — in this toy example the codeword lengths are whole numbers (1, 2, 3); it is only their average that is fractional, and averages of integers are routinely fractional. In general, code lengths themselves come out fractional too, and that is still meaningful: as part one argued, the information content of a long message is the sum of per-symbol contents, and arithmetic coding realises fractional per-symbol costs by encoding the whole message at once rather than symbol by symbol.

Going deeper, verified

• A Mathematical Theory of Communication — Claude Shannon (1948) · The source of log(1/p) as information content and of entropy as the cost floor; §6 is the part this page rests on.
• Information Theory, Inference, and Learning Algorithms — David MacKay (2003) · Free full text; chapters 4–6 build the same coding picture and give the "wrong codebook" cost as relative entropy, with proofs.
• torch.nn.functional.cross_entropy — PyTorch docs · Read the parameter table against the formula above: input is the model, target is reality, and the order is the reverse of H(p, q).
• scipy.stats.entropy — SciPy docs · The opposite convention (pk first), and a one-line way to check every number in this page's table.
• But what is information theory? (Compression is Intelligence Part 1) — 3Blue1Brown · The prerequisite: where the robot's optimal code and the entropy area diagram come from.

Exercises

1. Reproduce the 2.625 — In a notebook, define q = [1/2, 1/4, 1/8, 1/8] and p = [1/8, 1/8, 1/4, 1/2] and compute H(p), H(q), H(p, q), H(q, p) and D(p‖q) in bits. A good answer reports 1.75, 1.75, 2.625, 2.625, 0.875, states the identity H(p, q) = H(p) + D(p‖q) that ties them together, and explains in one sentence why the two cross-entropies coincide here without concluding that cross-entropy is symmetric.
2. Break the accident — Change a single entry so the relabelling symmetry is gone: keep q = [1/2, 1/4, 1/8, 1/8] and use p = [0.05, 0.05, 0.30, 0.60]. Recompute H(p, q) and H(q, p). A good answer shows the two numbers now differ, says which slot each distribution occupied in each case, and identifies which symbol contributed the largest single term to each sum.
3. Find the infinity, then tame it — Set q = [0.5, 0.5, 0.0, 0.0] against the same p and evaluate H(p, q) by hand; then evaluate H(q, p) by hand. A good answer gets ∞ for the first and a finite number for the second, explains the difference purely in coding terms (no codeword exists for a symbol that occurs, versus wasted probability on symbols that never occur), and then re-runs the first with q smoothed to [0.49, 0.49, 0.01, 0.01] to show what an implementation's clipping is buying you.