Scaling laws 1

Transcript: cleaned auto-captions with timestamps

By this point in CS336 you can build the machine — tokenizer and Transformer (A1), a distributed training stack and fast kernels (A2), and soon a pretraining corpus (A4). What you cannot yet do is choose. Lectures 3 and 4 answered "which architecture, which hyperparameters?" with the honest but unsatisfying "copy Llama," and that answer has a ceiling: it can reproduce the frontier, never move it. Lecture 9 supplies the missing method. It is the first lecture of the scaling arc, and its argument is that the frontier labs' apparent confidence — spend nine figures on one run and land it — is not bravado but extrapolation from a ladder of cheap experiments, with a well-understood statistical story underneath and a specific, well-documented failure mode on top.

Outline, with timestamps

• 00:04 — The setup, a hundred thousand H100s and one shot: you have all the pieces; which model do you actually run?
• 02:46 — Prehistory: Bell Labs 1993, Banko & Brill 2001, Kolachina 2012 — predicting a model's error before training it is a thirty-year-old idea.
• 08:08 — Hestness 2017 and the three-region curve; the Q&A on where scaling fails (inverse scaling, far out-of-distribution).
• 11:58 — What a data scaling law is: a map from dataset size n to excess error, linear in log–log.
• 15:45 — Two toy models: mean estimation gives slope −1; nonparametric regression gives slope −1/d.
• 21:12 — Data composition, repetition and selection: mixture changes the offset, not the slope; repeats decay after ~4 epochs.
• 26:11 — Model scaling: Transformer vs LSTM, Adam vs SGD, and the architecture sweep — all decided at small scale.
• 30:02 — Depth, width and aspect ratio; the embedding-parameter trap that bends an otherwise clean curve.
• 34:25 — Critical batch size: where data parallelism stops being free, and why the threshold moves with your loss target.
• 38:47 — Learning rate: fit a scaling law on the optimum, or reparameterize so there is nothing to fit (μP).
• 43:11 — The caution: perplexity scales beautifully, benchmarks do not.
• 47:00 — Joint data–model scaling laws: Rosenfeld's and Kaplan's functional forms, and how well they extrapolate.
• 50:48 — Chinchilla: why Kaplan was off (cosine schedules), and the three fitting methods.
• 61:01 — The method-3 forensics, train-optimal vs deploy-optimal, and a diffusion-LM replication.

Scaling laws did not start in 2020

Tatsu opens with a deliberate deflation. Public talk about scaling laws tends toward the eschatological — lines go straight forever, therefore superintelligence — and he wants the class to see an ordinary object: an empirical curve fit, of a kind statisticians have drawn since before deep learning existed. The oldest paper he can find doing the modern thing is a 1993 NeurIPS paper out of Bell Labs by Cortes, Jackel, Solla, Vapnik and Denker. Its motivation is verbatim the 2025 motivation — training classifiers on large databases is expensive, so predict which ones are worth training — and so is its functional form: an irreducible error term plus a polynomially decaying term, fit on small runs and extrapolated. Banko and Brill (2001) made the same shape famous in NLP with the argument that has echoed ever since, and by 2012 Kolachina and colleagues were arguing explicitly about which parametric family (Pow3, Pow4, …) actually predicts a learning curve.

The paper that most repays reading is Hestness et al. 2017, done at Baidu, where "neural scaling law" becomes a research program rather than an observation. Two things from it survive intact. The first is the three-region picture of a learning curve: a small-data region where the model is near chance and nothing is predictable, a middle power-law region where error falls as a straight line in log–log, and an asymptotic region approaching the irreducible error of the model class. Everything anyone fits is the middle region; the other two are where extrapolations die. The second is that Hestness already reasoned about emergence (predictions are hopeless while a model sits at random performance, and it can leave that regime abruptly), about compute as the axis that matters, and about quantization — if capability is a predictable function of compute, trading precision for compute becomes a rational purchase. A lot of what reads as 2023 discourse was legible in 2017 to anyone looking at the plots.

Why a straight line on a log–log plot?

Monotonicity is free — more data, less error. The claim that costs something is the functional form: linear in log–log means a polynomial relationship, error ∝ n^−α, and there is no a priori reason a network should obey one. The two toy models here are the best twenty minutes in the lecture: they make the power law feel inevitable, then explain why the exponent is small.

Start with the simplest estimation problem there is. Draw x₁…xₙ ~ N(μ, σ²) and estimate the mean by the sample average. The expected squared error is σ²/n; take logs and log(Error) = −log n + 2 log σ. That is a scaling law — a straight line, slope −1, intercept set by the noise level. No deep learning required; polynomial decay of estimation error is the normal condition of statistics. Classical parametric models mostly land at 1/n and agnostic learning at 1/√n, so the naive prediction is that fitted slopes should be round numbers near 1 or 0.5.

They are not. The exponents Tatsu reads off the Hestness plots are roughly 0.13 for machine translation, 0.3 for speech, and 0.095 for language modeling — an order of magnitude shallower than theory suggests. The second toy model explains the gap. Neural networks are not estimating a mean; they are flexible enough to approximate arbitrary functions, so the right comparison is nonparametric estimation. Put n points uniformly in a 2-D box, let yᵢ = f(xᵢ) + noise for a smooth f, and estimate f by chopping the space into boxes and averaging within each. With √n boxes, each box holds about √n samples, and the error goes as 1/√n. In d dimensions the same accounting gives Error ≈ n^−1/d — a straight line in log–log with slope −1/d. Flexibility buys you the power law and costs you the exponent, and the exponent is now telling you something about the effective dimension of the problem rather than about the estimator.

This is the seed of the intrinsic-dimensionality account (Bahri et al.), which argues the fitted slope is essentially a readout of the data's intrinsic dimension. Tatsu shows the fit and immediately backs off it: estimating the intrinsic dimension of something like CIFAR is about as hard as modelling the data, so the story is suggestive rather than settled — the lecture's stance in miniature, take the regularity seriously and hold the explanation loosely.

Data scaling as an engineering instrument

Once you believe the power law you can use it to make decisions, not just predictions. Three uses come up. Composition: Kaplan's out-of-distribution figures show that changing the data distribution moves the offset of the line, not its slope — a licence to compare corpora at small scale and trust the ordering at large scale. Repetition: with finite unique tokens, the data-constrained scaling work reparameterizes the law in terms of effective data given unique tokens and a repetition count, and finds returns collapsing after roughly four epochs. Selection under finiteness: combining the two, the CMU data-filtering work argues curation cannot be compute-agnostic — as your budget grows, aggressive filtering forces you to re-read a small high-quality pool until repeats are worth less than fresh, lower-quality tokens, so the optimal filtering threshold is itself a function of scale.

One caveat matters for the assignment: when you vary n alone, model size is held far above what the data can saturate, precisely so you never enter the asymptotic region. A student asks this directly at 24:30; the answer is one big model for the whole data-scaling curve. Every one-variable scaling law is implicitly conditioned on the other variables being generous.

Model scaling: turning design questions into small-model experiments

The second half of the lecture is where scaling laws stop being a phenomenon and become a procedure.

"We're going to train a bunch of small models, we're going to learn a lot of things from those small models, and then we're going to extrapolate them back up to bigger models."— Tatsunori Hashimoto, 02:13

The mechanism that makes this legitimate is a specific empirical shape: plot compute against loss for two competing design choices and the curves are usually parallel and non-crossing, separated by a constant factor. If A is a fixed multiple of B's compute-efficiency at 10¹⁷ FLOPs, it stays that multiple at 10²³. That is what licenses deciding at small scale. Kaplan's Transformer-vs-LSTM figure is the canonical instance — LSTMs sit above Transformers by a constant compute penalty at every depth tested (Tatsu waves at "maybe 15×" as an illustration of reading such a gap, not as a measured constant). Tay et al.'s wider sweep runs the same experiment across many architectures against a Transformer baseline; the only ones that reliably out-scale it are gated linear units and mixtures of experts — not coincidentally, exactly what shipped in production models. Optimizer choice behaves the same way in the Hestness data: Adam beats SGD by a constant factor across the range, so you can settle it small.

Depth and width are more interesting because the answer is "it barely matters." One layer versus two is a large difference; past that, the curves for different layer counts crowd together and returns are thin below ~10⁷ parameters. Kaplan's aspect-ratio slices at 50M, 274M and 1.5B parameters show the same shape at every scale, with a wide flat basin — roughly 10 to 100 for width-to-depth is near-optimal — the scaling-law version of the rule of thumb from the architecture lecture. The trap here is parameter counting: include embedding parameters and the clean log-linear curve bends; exclude them and it straightens. Embeddings do not behave like compute-carrying parameters, and the question comes back harder for MoE, where the literature has to invent a dense-equivalent parameter count before anything is comparable.

The two knobs that move with scale

Architecture and optimizer transfer. Batch size and learning rate do not, and they are coupled to each other.

Batch size has a threshold, the critical batch size. Below it, doubling the batch is about as good as taking two gradient steps — a free lunch, since you get the optimization progress of two steps with the parallelism of one. Above it, the extra samples are no longer cancelling gradient noise, the curvature term dominates, and you are buying steps you do not get. The non-obvious fact is that the threshold depends on your loss target: the lower the loss you are aiming at, the larger the critical batch size. That is why real training reports ramp the batch mid-run — Llama 3 does exactly this. Kaplan's compute-vs-batch analysis then delivers the good news that as you scale compute you can absorb most of it as larger batches at roughly constant step count, which is what data parallelism wants to hear. A student pushes on the intuition at 42:03 and Tatsu's answer is a denoising one: a more sensitive target demands more precise gradients, and batch size and learning rate move inversely.

Learning rate has two strategies. The standard-parameterization one accepts that the optimum drifts — wider models want smaller learning rates, with 1/width as the folk rule — finds the minimum of the loss-vs-LR curve at several scales, and fits a scaling law to the optimum itself. The more ambitious one changes the parameterization so there is nothing to extrapolate: scale initialization variance, per-layer learning rates and forward-pass multipliers as functions of width so the optimal LR is width-invariant, tune once on the smallest model, transfer. That is μP (Yang et al. 2022). Tatsu notes several labs are chasing variants — including, without endorsement, a "MetaP" Meta claims for Llama 4 whose details he does not know — and that "tune once, transfer exactly" is more optimistic than practice. Lecture 11, Scaling laws 2, goes deeper.

Downstream scaling is not loss scaling

This is the lecture's sharpest warning and the one most often ignored. Tay et al. give the demonstration: across many architecture and hyperparameter variants, parameter count (as a compute surrogate) versus negative log perplexity is a near-perfect line — depth, width and the rest wash out, only total compute matters. Take those same models to a downstream benchmark and the line dissolves; some architectures are simply better at the task at equal perplexity. The same pattern recurs for state-space models, which scale beautifully on loss and then underperform on in-context learning and question answering.

"It's important to not take this perplexity scaling as the same thing as downstream scaling, and you want to be a little bit cautious whenever you're doing these kinds of analyses."— Tatsunori Hashimoto, 45:22

Stripped of caveats, the design procedure is three steps: train a handful of small models spanning at least a couple of orders of magnitude of compute; verify and fit a log–log linear relationship; pick the hyperparameter the fit prefers. Where the slopes match and only the offsets differ, you can shortcut even further — the small-model ranking transfers directly. Learning rate is the standing exception.

Chinchilla, and what "compute-optimal" leaves out

The headline application is the allocation question. For a fixed compute budget, do you want a big undertrained model or a small well-trained one? Both extremes are wasteful, so there is an interior optimum, and finding it requires a joint law in data and model size. Two were proposed nearly simultaneously: Rosenfeld et al.'s Error = n^−α + m^−β + C — a data term, a model term, an irreducible floor — and Kaplan et al.'s functionally similar form without the constant. Tatsu is candid that these forms are ad hoc, pulled out of a hat, and equally candid that they fit alarmingly well: Rosenfeld's 3-D loss surface passes through the runs almost exactly, and fitting on the small-model/small-data quadrant alone predicts the large quadrant accurately on both ImageNet and WikiText.

Kaplan's allocation answer nonetheless turned out to be wrong, and the reason is a piece of experimental hygiene. Everyone trains with cosine learning-rate schedules, and a cosine schedule cannot be truncated: a run stopped halfway is not equivalent to a run planned to end there, because it never got its cooldown. Read intermediate checkpoints off long runs and you systematically mis-measure what a shorter run would have achieved. Chinchilla (Hoffmann et al. 2022) redid the analysis with schedules matched to budgets, three ways:

All three sit near equal scaling of parameters and data, against Kaplan's a = 0.73, b = 0.27. Projected to Gopher's compute budget (5.76 × 10²³ FLOPs), methods 1, 2 and 3 predict optimal model sizes of 67B, 63B and 40B parameters; Chinchilla itself was trained at 70B on 1.4T tokens and beat the 280B Gopher trained on the same budget. Hence the "20 tokens per parameter" number everybody quotes.

Method 3's disagreement bothered Tatsu's group for years, and the resolution is the lecture's best story. Epoch AI could not get the original run data, so they extracted the points from the published figures with a forensic tool, reproduced the fit, and found the residuals were not zero-centred — a curve-fitting error, not a data error. Refit properly, method 3 lands on methods 1 and 2.

"Usually replication sort of disproves things, but in this case actually the replication just showed that the original result was correct all along, which is I think a pretty cool result."— Tatsunori Hashimoto, 61:01

Train-optimal is not deploy-optimal. The last move is that Chinchilla answers a question you may not be asking: it optimizes loss per unit of training compute. A deployed model spends most of its lifetime compute on inference, so the right objective includes serving cost — which pushes you to deliberately "over-train" a smaller model well past its Chinchilla point. The drift in published ratios is stark:

The rule is amortization: the more inference you expect to serve, the more it pays to move cost forward into pretraining. Tatsu points at Qwen 3, released the day before this lecture, as the then-current extreme — citing ~30T tokens from memory, where the technical report says roughly 36T. He closes with evidence that none of this is Transformer-specific: his group ran the IsoFLOP playbook on diffusion language models and got Chinchilla-shaped curves, offset by a constant from the autoregressive ones. The methodology is cheap enough that "run your own IsoFLOP sweep" is a reasonable first move on any new model class.

What you build with this

This lecture opens Assignment 3: Scaling (Spring 2025 handout PDF — note that the repository's main branch has since rolled forward to the 2026 version, which restates the same problem in B200-hours against a hosted training API). A3 is the only assignment in the course where you do not train the model you are reasoning about. You are given a target budget of 10¹⁹ FLOPs and a separate, hard-capped experiment budget of 2 × 10¹⁸ FLOPs — 20% of the target — to spend querying a training API that returns final training loss for a requested set of hyperparameters and FLOP count.

Two problems. chinchilla_isoflops (5 points) has you implement Chinchilla method 2 offline on synthetic runs in data/isoflops_curves.json: group runs by compute budget, take the minimum-loss run in each profile, fit N_opt ∝ C^a and D_opt ∝ C^b, and extrapolate to 10²³ and 10²⁴ FLOPs. scaling_laws (50 points) is the real thing: design a sweep that fits inside 2 × 10¹⁸ FLOPs, fit your own law, and predict the compute-optimal model size, its hyperparameters and its final loss at 10¹⁹ FLOPs — with part of the grade riding on how good the predicted configuration actually is. Batch size is restricted to 128 or 256 to keep MFU sane. The binding constraint is experiment design: every query you spend is gone, so the ladder of compute scales and the model sizes within each rung have to be planned before you start. Everything in this lecture about the power-law region, non-embedding parameter counts, and cosine schedules is directly load-bearing for that plan.

Supporting materials, verified

• Scaling Laws for Neural Language Models — Kaplan, McCandlish, Henighan, Brown, et al. (2020) · The lecture's backbone: compute/data/parameter power laws, LSTM-vs-Transformer, aspect-ratio slices, embedding-parameter caveat, batch-size scaling. Tatsu calls it a gold mine and recommends reading it whole.
• Training Compute-Optimal Large Language Models — Hoffmann et al. (2022) · Chinchilla: the three fitting methods, the ~20 tokens/parameter ratio, and the demonstration that a 70B model beats a 280B one at equal training compute.
• Deep Learning Scaling is Predictable, Empirically — Hestness et al. (2017) · The three-region learning curve, cross-domain power laws (MT, speech, LM) and the Adam-vs-SGD constant-factor gap.
• A Constructive Prediction of the Generalization Error Across Scales — Rosenfeld, Rosenfeld, Belinkov, Shavit (2020) · The joint data–model functional form n^−α + m^−β + C and the small-quadrant-to-large-quadrant extrapolation test.
• Learning Curves: Asymptotic Values and Rate of Convergence — Cortes, Jackel, Solla, Vapnik, Denker (1993) · The Bell Labs paper Tatsu nominates as the first scaling-law paper: irreducible error plus a polynomial term, fit small and extrapolated.
• Scaling to Very Very Large Corpora for Natural Language Disambiguation — Banko & Brill (2001) · The log-linear data-scaling plot that made "more data beats better algorithms" an NLP argument.
• Prediction of Learning Curves in Machine Translation — Kolachina, Cancedda, Dymetman, Venkatapathy (2012) · Early systematic comparison of candidate parametric families for learning curves.
• Explaining Neural Scaling Laws — Bahri, Dyer, Kaplan, Lee, Sharma (2021) · The intrinsic-dimensionality account of why exponents are what they are; the theory Tatsu presents and then hedges.
• Model Performance Scaling with Multiple Data Sources — Hashimoto (2021) · The lecturer's own work on predicting performance under data mixtures; the "composition moves the offset" claim.
• Scaling Data-Constrained Language Models — Muennighoff et al. (2023) · Scaling laws under repetition, with effective-data and unique-token terms; the ~4-epoch diminishing-returns result.
• Scaling Laws for Data Filtering — Data Curation cannot be Compute Agnostic — Goyal, Maini, Lipton, Raghunathan, Kolter (2024) · The CMU work on trading repetition of high-quality data against fresh lower-quality data as a function of budget.
• Scaling Laws vs Model Architectures — Tay et al. (2022) · The cross-architecture sweep behind both the "only GLU and MoE reliably out-scale the Transformer" slide and the downstream-unpredictability warning.
• Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (μP) — Yang, Hu, et al. (2022) · Width-aware initialization and per-layer learning-rate scaling so the optimal LR transfers from small models.
• An Empirical Model of Large-Batch Training — McCandlish, Kaplan, Amodei et al. (2018) · The gradient-noise-scale theory behind the critical batch size.
• Chinchilla Scaling: A Replication Attempt — Besiroglu et al. (2024) · The Epoch AI forensics: data extracted from the published figures, non-zero-centred residuals in method 3, and a refit that agrees with methods 1 and 2.
• Likelihood-Based Diffusion Language Models — Gulrajani & Hashimoto (2023) · The closing case study: IsoFLOP analysis reproduced on diffusion LMs, giving Chinchilla-shaped curves offset from the autoregressive ones.
• Language Models are Few-Shot Learners (GPT-3) — Brown et al. (2020) · The pre-Chinchilla allocation point, at ~2 tokens per parameter.
• Inverse Scaling: When Bigger Isn't Better — McKenzie et al. (2023) · Field map extra. The Inverse Scaling Prize results Tatsu references in Q&A on where scaling stops being monotone.
• The Llama 3 Herd of Models — Grattafiori et al. (2024) · Field map extra. A production run that ramps batch size mid-training and sits at ~215 tokens/parameter — both slides made concrete.
• Qwen3 Technical Report — Qwen Team (2025) · Field map extra. Released the day before this lecture; the ~36T-token pretraining budget Tatsu cites as the current extreme of over-training.
• Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data — Villalobos et al. (2022/2024) · Field map extra. The quantitative version of the "are we running out of internet?" question that motivates the repetition and selection laws.

Exercises

1. IsoFLOP fit from scratch code — Implement Chinchilla method 2 on data/isoflops_curves.json from the assignment repo. Steps: (1) group runs by compute_budget; (2) for each budget take the minimum-loss run, and separately fit a quadratic in log N and take its vertex; (3) fit N_opt ∝ C^a and D_opt = C/(6N_opt) ∝ C^b by least squares in log space; (4) extrapolate to 10²³ and 10²⁴ FLOPs; (5) plot both fits with the points. A good answer reports a and b near 0.5, states how much the argmin-vs-quadratic choice moves the answer, and says which compute budgets dominate the fit.
2. Budget the sweep before you spend it code — Before touching the A3 training API, write a planner that proposes a set of (compute budget, model size) queries totalling ≤ 2 × 10¹⁸ FLOPs. Steps: (1) pick 4–6 IsoFLOP rungs spanning at least two orders of magnitude below 10¹⁹; (2) put 4–6 model sizes on each rung, bracketing the expected optimum from the Chinchilla ratio; (3) cost the plan and check it fits; (4) simulate the fit on synthetic losses to confirm the design is identifiable. A good answer justifies where the budget goes — more rungs (better extrapolation) versus more points per rung (better minima) — and states what it would drop first.
3. Break your own scaling law — Take any published loss-vs-compute fit and construct three cases where extrapolating it should fail, one per failure mode in the lecture: fitting inside the near-chance region, fitting with embedding parameters counted, and fitting on truncated cosine-schedule checkpoints. For each, say what the fitted curve looks like (it will look fine) and what diagnostic would catch it.
4. Loss versus benchmark — Using any public model family with released intermediate checkpoints, plot validation loss against compute and a downstream benchmark against compute on the same x-axis. A good answer identifies where the two disagree, and states which decisions you would still be willing to make from the loss curve alone.