Distillation: training on a distribution instead of an answer

Transcript: this stretch, timestamped

P13 finished the hard part of the argument: if you want a per-token loss whose average is minimised only when the model's output distribution matches the statistics in the data, your hand is forced into the negative logarithm, and the average that falls out is literally the cross-entropy formula. That derivation left cross-entropy looking slightly ceremonial, because in ordinary pre-training the p you take cross-entropy against is a spike on one token and the sum collapses to a single term. This page is where Grant shows the formula being used with both slots genuinely populated — that is distillation — and then steps back to survey the whole video before handing off to Part 3. P15 picks up the last footnote, KL divergence, which is exactly the gap this page keeps circling.

Outline, with timestamps

• 26:13 — Setup: a variant of training where the cross-entropy formula is used "more explicitly", and each example tugs the model more softly.
• 26:32 — Why anyone wants this: you have a big model that works, and you want that behaviour at a much cheaper inference cost.
• 27:02 — The swap: compare the student's distribution not to the true next token, but to the teacher's full distribution at the same position.
• 27:34 — The chess analogy, and why one distilled example can do the work of millions of one-hot examples.
• 28:06 — Zoom out, part one: cross-entropy arose naturally from compression, and generalised into a way to measure how one set of patterns differs from another.
• 28:37 — The honest admission: the second half's derivation came from constrained optimisation, not compression — but π always hides a circle.
• 29:10 — The bridge to Part 3: turn a predictor into a compressor whose bit count matches the model's own information content.
• 29:42 — Payoff promised: cross-entropy loss is training the model to be the best possible text compressor.

What one-hot pre-training is actually optimising 26:13

Write the per-token loss the way P13 left it. The model, given a prefix, emits a distribution q over the whole vocabulary; the target is a distribution p over the same vocabulary; the loss is

H(p, q)  =  − Σ  p(x) · log q(x)
              x ∈ V

In standard pre-training p is one-hot: p(x) = 1 for the token y that actually appeared in the corpus, and 0 for the other fifty-odd thousand entries of the vocabulary V. Every term but one is multiplied by zero, and what survives is

H(p, q)  =  − log q(y)

which is the plain surprisal of the true token. That is the whole of pre-training. But notice what the collapse costs you. The formula has room for a value on every token in the vocabulary, and you are using one slot of it. Whatever opinion the model formed about the other 50,000 tokens is left completely unconstrained by this example. The only instruction the gradient carries is push mass onto y, take it off everything else, proportionally. Nothing distinguishes a near-miss from an absurdity.

The reason this works at all is the one Grant established at 22:23: averaged over the whole corpus, the one-hot targets reconstruct the real distribution, because the frequency with which each name follows "my name is" is the p you wanted. You never write the distribution down; you pay for it in repetitions.

The swap: put the teacher in the p slot 27:02

Distillation changes exactly one thing. You have a large model that already performs well — call its output at this position p_T — and a small student q_S you want to make cheap enough to serve. Run both over the same text. At each position, instead of comparing the student to the corpus's one-hot answer, compare it to the teacher's full distribution:

L  =  H(p_T , q_S)  =  − Σ  p_T(x) · log q_S(x)
                          x ∈ V

Same formula, different p. Now no term vanishes. If the teacher spreads 0.20 on James, 0.15 on Maria, 0.02 on Xiomara and 10⁻⁷ on bulldozer, all of that is a constraint the student is being pushed to satisfy at once. And it is available from a single pass over that piece of text. Where one-hot pre-training needs to see "my name is ___" completed thousands or millions of times before the empirical frequencies approximate a real name distribution, distillation hands you the answer in one shot, because the teacher already did the averaging.

"As an analogy, it's the difference between trying to learn chess just by watching a game versus having someone better than you talk through all the possible good moves and how heavily they would weight them at each new turn."— Grant Sanderson, 27:34

That analogy is doing more work than it looks. Watching moves gives you the argmax of a strong policy. Hearing the weighting gives you the policy.

Dark knowledge: the signal lives in the wrong answers 27:34

The rest of this section is my addition — Grant states the idea but does not name the literature. The classic reference is Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network" (2015, arXiv:1503.02531; presented at the NIPS 2014 Deep Learning Workshop), building on Buciluǎ, Caruana and Niculescu-Mizil's 2006 "Model Compression". Their argument for why the extra terms carry so much is sharper than "more numbers is more information".

A trained classifier's probabilities on the incorrect classes are not noise. They are a learned similarity structure. The paper's example: an image of a BMW may have only a tiny chance of being called a garbage truck, but that mistake is still many times more probable than calling it a carrot. The ratio between two near-zero probabilities encodes which things the model considers confusable — which is precisely the generalisation behaviour you want to transfer, and precisely what the hard label cannot express. Hinton called this the dark knowledge in a model; the term comes from his talks about this work rather than from the paper itself.

The catch is that this information sits in ratios of very small numbers, and cross-entropy weights each term by p_T(x). If the teacher is confident, a probability of 10⁻⁶ versus 10⁻⁹ is an enormous ratio contributing an utterly negligible amount to the loss. The rich part of the signal is present and inaudible.

Temperature: turning up the contrast 27:02

Also my addition; the video does not cover temperature. The fix in the 2015 paper is to soften both distributions before comparing them, by dividing the logits z_i by a temperature T inside the softmax:

q_i(T)  =  exp(z_i / T)  /  Σ_j exp(z_j / T)

T = 1 is the ordinary softmax. T > 1 flattens the distribution toward uniform, which raises the weight on the small probabilities so their ratios start to matter to the loss; T < 1 sharpens it. Concretely, for logits (4, 2, 1, 0, −1):

Two details people get wrong. First, the same T must be applied to the student's softmax during training, and the student reverts to T = 1 at deployment. Second, the gradient rescaling. The paper derives the gradient of the soft-target cross-entropy with respect to the student's logit z_i as (1/T)·(q_i − p_i), and in the high-temperature limit — with logits zero-meaned per example — this approximates (z_i − v_i) / (N·T²), where v_i are the teacher's logits. So the soft-target gradients scale as 1/T², and when you mix a soft-target term with a hard-label term you must multiply the soft term by T² to keep their relative contributions stable as you tune T. That T² factor is the single most commonly dropped line in reimplementations. (A pleasant corollary from the same derivation: in the high-T limit distillation reduces to matching logits with a squared error, which is what the 2006 model-compression work did directly.)

What the swap buys, information-theoretically — and what it does not 28:06

Here is the cleanest way I know to say what is going on, and it is worth writing out because it also tells you the limits. Suppose the true conditional distribution at this position is p, and the corpus hands you a single token y drawn from it. The one-hot loss −log q(y) is a random variable, and its expectation is

E          [ − log q(y) ]  =  − Σ p(x)·log q(x)  =  H(p, q)
 y ~ p                          x

So the one-hot loss is an unbiased estimator of the cross-entropy you actually wanted — but a one-sample one, with real variance. Take a five-token toy vocabulary where the true p and the student's q are:

The full-distribution loss is H(p, q) = 1.9147 bits. The entropy floor is H(p) = 1.8309 bits, so the gap — the KL divergence P15 is about to name — is 0.0839 bits. The one-hot loss has that same mean of 1.9147, and a standard deviation of 0.8377 bits: on any given example it is off by roughly half its own value. Every one of those errors is a wrong gradient, and pre-training only survives it by averaging over an astronomical number of tokens.

Distillation replaces the sample with the distribution. If the teacher were the true p, the variance from this source would be exactly zero — you would compute the quantity instead of estimating it. That is the real content of "richer signal per example", and it is why Hinton et al. observed that soft targets let a small model be trained on much less data and with a much higher learning rate.

And now the honest half. The teacher is not p. It is a model, with its own biases and confident errors, and the student is being trained to reproduce them faithfully. Two consequences follow directly from the formula. First, minimising H(p_T, q_S) over q_S is minimised exactly at q_S = p_T — so on the transfer set the student's target is being the teacher, never beating it. Second, the loss floor is H(p_T), the teacher's entropy, not the entropy of language: a perfect student still reports a nonzero loss equal to the teacher's own uncertainty. What distillation buys is variance, not truth. It gives you a low-noise estimate of a slightly wrong target, in exchange for the high-noise estimate of a correct one — and empirically that trade is overwhelmingly worth it, which is why the technique is everywhere. In practice you usually hedge by keeping a hard-label term in the mix, so the student is still anchored to the corpus where the two disagree.

Two verified examples of the technique at production scale: DistilBERT (Sanh et al., 2019) distils BERT during pre-training to 40% smaller while retaining 97% of its language-understanding score and running 60% faster; and the Gemma 2 report states that the 2B and 9B models were trained with knowledge distillation "instead of next token prediction", citing Hinton et al. (2015). Modern practice has moved on in the details — much LLM distillation now happens at the sequence level, or on text the teacher generated rather than on a fixed corpus — but the object being minimised is still the cross-entropy on this page.

Zooming out: two roads to the same formula 28:37

Grant then stops and does something a lecturer should do more often: he flags that the video has produced the same formula twice by routes that do not obviously touch. In the first half, cross-entropy fell out of compression — you build an optimal code for one distribution and ask what it costs on another, and −Σ p(x)·log₂ q(x) is the answer in bits per symbol. In the second half it fell out of constrained optimisation — you demand a loss minimised only at q = p, apply a Lagrange multiplier, and the logarithm is forced. Those are different fields. Nothing in the second derivation mentions codes.

"In the same way that whenever you see pi show up in math, even when it seems unrelated to circles, there's always going to be some connection to circles hiding there, I think whenever you see cross-entropy pop up, there will be a way to connect it to compression."— Grant Sanderson, 28:37

The hinge that will close the loop is already on the table: the pre-training loss was described from the start as the average information per token from the model's perspective. "Information", here, is measured in the units of a code — log₂(1/q(x)) is a length. If you can actually build a compressor that spends that many bits per token, then the loss stops being an analogy about codes and becomes a literal file size. 29:10

That construction is Part 3, and Grant flags it as genuinely surprising that it is possible at all: a scheme that turns any predictor into a compressor whose output length matches the predictor's own information content, including the fractional bits that Huffman coding cannot spend. Once you have it, "minimise cross-entropy loss" and "be the best possible text compressor" are the same instruction, and the slogan compression is intelligence becomes a claim you can actually evaluate rather than admire. 29:42

Where people get stuck

Four confusions specific to this stretch:

• Which model goes in which slot. H(p, q) = −Σ p·log q is not symmetric, and the roles are not interchangeable. The teacher is p — the weights, outside the log. The student is q — inside the log, and the only thing with gradients. Swap them and you are optimising the teacher's fit to the student, which is both wrong and, with a frozen teacher, a no-op.
• "Shouldn't the loss be KL, since KL is the distance?" This is Grant's own parting puzzle. The answer for a frozen teacher: D(p_T‖q_S) = H(p_T, q_S) − H(p_T), and the subtracted term does not depend on the student's parameters at all. The two losses differ by a constant, so their gradients are identical and training is unchanged. KL is the more interpretable number — it is zero at a perfect match instead of bottoming out at the teacher's entropy — and cross-entropy is the cheaper one, because you never have to evaluate H(p_T). If the teacher is not frozen (online or mutual distillation), the constant is no longer constant and the choice starts to matter. P15 unpacks the divergence itself.
• "One-hot means the target is the truth." No. In pre-training p is a single draw from the true conditional, not the conditional. Both pre-training and distillation are approximating the same ideal cross-entropy against the real distribution of language; they just use estimators with very different bias–variance profiles. Seeing this clearly is what stops distillation from looking like a hack.
• Temperature is not a smoothing hyperparameter you can set and forget. Raising T changes the loss surface and shrinks the soft-target gradients by 1/T². Without the compensating T², sweeping T in a mixed loss is silently also sweeping the relative weight of your two objectives, and you will attribute the result to temperature. Also note the paper's finding that the best T depends on student capacity: in their MNIST experiment, students with 300+ units per layer were insensitive to T above 8, but a 30-unit student did much better at T in the 2.5–4 range — a very small student is better off ignoring the deep tail.

Going deeper, verified

• Distilling the Knowledge in a Neural Network — Hinton, Vinyals & Dean (2015) · The source for everything on this page beyond Grant's four minutes: temperature, the T² rescaling, the logit-matching limit, and the MNIST result where a distilled 800-unit net drops from 146 to 74 test errors against a 67-error teacher.
• Model Compression — Buciluǎ, Caruana & Niculescu-Mizil, KDD 2006 · The precursor Hinton et al. build on; transfers an ensemble into one small net by matching outputs. (The ACM DOI page returns 403 to scripted requests; this is the authors' copy.)
• Sequence-Level Knowledge Distillation — Kim & Rush (2016) · Where the idea stops being per-token: distilling on the teacher's generated sequences rather than its per-position distributions, which is much closer to how LLM distillation is done now.
• DistilBERT, a distilled version of BERT — Sanh, Debut, Chaumond & Wolf (2019) · A worked, widely reproduced case of distilling during pre-training rather than per task; the triple loss shows how a soft-target term is combined with the ordinary one in practice.
• Gemma 2: Improving Open Language Models at a Practical Size — Gemma Team (2024) · Evidence the technique is load-bearing at frontier scale: the 2B and 9B models were trained with knowledge distillation in place of next-token prediction.

Exercises

1. Reproduce the variance argument — With p = (0.50, 0.25, 0.15, 0.07, 0.03) and q = (0.40, 0.30, 0.10, 0.15, 0.05), compute H(p, q), H(p), D(p‖q), and the standard deviation of the one-hot loss −log₂ q(y) for y ~ p. You should recover 1.9147, 1.8309, 0.0839 and 0.8377 bits. A good answer also states how many independent one-hot samples you would need before the sample mean's standard error drops below 0.01 bits — the arithmetic is (0.8377/0.01)² ≈ 7,000 — and connects that number to the claim that distillation needs far less data.
2. Feel the temperature — For logits (4, 2, 1, 0, −1), tabulate the softmax and its entropy for T ∈ {0.5, 1, 2, 4, 8, 32}, and plot the ratio of the 4th to the 5th probability against T. A good answer explains why that ratio is monotone in 1/T and what happens to it as T → ∞, then numerically checks the paper's claim by computing ∂L/∂z_i by finite differences at two temperatures and confirming the magnitudes fall roughly as 1/T².
3. Show the gradients are the same — Prove that for a frozen teacher, ∇_θ H(p_T, q_S) = ∇_θ D(p_T‖q_S), where θ are the student's parameters. Then answer the harder half: name one concrete situation in which the two losses genuinely diverge in behaviour, and say what changes. (Hint: what has to be true of the teacher for H(p_T) to be a constant?)