Automating backward: per-op closures, topological order, and the accumulation bug

Transcript: this part, with timestamps

This is the hinge of the lecture. Up to 69:02 backpropagation has been a thing Karpathy does — an activity, performed by a human who understands calculus, typing x1.grad = w1.data * x1w1.grad into a notebook cell one node at a time. By 87:05 it is a thing the data structure does to itself. Nothing about the mathematics changes across those eighteen minutes; every formula that appears was already derived by hand in P3. What changes is where the knowledge lives. The chain rule for an addition stops being something in Karpathy's head and becomes a Python function stored on the node that the addition produced. That relocation is what "autograd engine" means, and it is roughly forty lines of code.

Outline, with timestamps

• 69:02 — Chapter: implementing the backward function for each operation. The manual approach is declared dead; the per-node chain-rule step moves onto the node itself, defaulting to lambda: None for leaves.
• 70:40 — The closure for addition. Defined inside __add__ after out exists, so it can see self, other and out; local derivative 1.0, so the gradient is copied to both inputs.
• 72:11 — The closure for multiplication. Local derivative of a product with respect to one factor is the other factor's data; the two operands swap.
• 72:42 — The closure for tanh. Local derivative 1 − t², where t is the output the forward pass already computed and the closure still holds.
• 74:14 — Calling them by hand, and the base case. o.grad must be set to 1.0 first, or every closure multiplies by zero.
• 75:17 — The typo he leaves in. out._backward = _backward() stores None; the fix is to drop the parentheses.
• 77:32 — Chapter: backward for a whole expression graph. The ordering constraint stated: a node's own gradient must be final before it is allowed to push anything backwards.
• 78:24 — Topological sort, and build_topo. The graph is a DAG, so a linear order exists in which every edge points forward; the depth-first builder appends a node only after all of its children, and reversing that list is a legal backward schedule.
• 81:26 — Folded into Value.backward(). Build the order, seed self.grad = 1.0, loop over reversed(topo) calling _backward().
• 82:28 — Chapter: the bug when one node is used more than once. b = a + a gives a.grad = 1; then a second, less obvious example with a shared input.
• 85:03 — The fix. The multivariate chain rule adds contributions along every path, so every assignment to grad becomes +=.
• 86:38 — Cleanup. The scratch cells are deleted, the class survives, and the next question is whether tanh had to be a primitive at all.

One closure per operation

The idea is small enough to state in one sentence: every node that was produced by an operation should carry a function that knows how to push its own gradient into the gradients of the things it was built from. Not "how to compute a derivative" in the abstract — how to push this node's gradient into these specific inputs. That specificity is why a closure is the right tool. When __mul__ runs, it has three objects in scope that matter: self, other, and the freshly built out. A function defined inside __mul__ captures all three by reference, so it can be called later — much later, in a completely different part of the program — and still be talking about the same three nodes.

The constructor gains a default so that every node has the attribute whether or not anything meaningful lives there:

self._backward = lambda: None

That default is not a placeholder to be filled in later; for a leaf it is the correct and final answer. x1, w1, b are inputs. They have no children. There is nothing for them to propagate into, so their _backward does nothing, forever. The uniformity matters more than it looks: it means the driver loop later can call node._backward() on every node in the graph without ever checking what kind of node it is.

Then the three operations each grow four lines:

def __add__(self, other):
  out = Value(self.data + other.data, (self, other), '+')

  def _backward():
    self.grad += 1.0 * out.grad
    other.grad += 1.0 * out.grad
  out._backward = _backward

  return out

def __mul__(self, other):
  out = Value(self.data * other.data, (self, other), '*')

  def _backward():
    self.grad += other.data * out.grad
    other.grad += self.data * out.grad
  out._backward = _backward

  return out

def tanh(self):
  x = self.data
  t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1)
  out = Value(t, (self, ), 'tanh')

  def _backward():
    self.grad += (1 - t**2) * out.grad
  out._backward = _backward

  return out

Read each closure body as one instance of the chain rule, written in the same shape every time: input's gradient gets the local derivative of the output with respect to that input, times the output's gradient. The second factor, out.grad, is the part that arrives from downstream — it already encodes everything that happens between this node and the final scalar. The first factor is the only new information this node contributes.

For addition, the local derivative of a + b with respect to a is 1, and with respect to b is also 1. So a plus node does nothing but hand its gradient, unchanged, to both children. That is why Karpathy keeps calling additions the easy ones: a plus is a router. For multiplication the local derivative with respect to one factor is the other factor's value, so a times node swaps and scales — and note it reads other.data, the forward value, not other.grad. Mixing those two up is the single most common typo in a hand-written engine. For tanh the local derivative is 1 − tanh(x)², and the closure does not recompute a tanh: it captured t, the number the forward pass already produced, and uses that. This is the general trick in every real autograd — the backward pass is cheap partly because the forward pass left useful intermediates lying around in closures.

One line is easy to skim past. out._backward = _backward stores the function object; it does not call it. Karpathy gets this wrong live, writes out._backward = _backward(), and hits TypeError: 'NoneType' object is not callable a few minutes later when the driver tries to invoke the None that the call returned. He keeps the mistake in the video on purpose.

i thought about redoing it but i figured i should just leave the error in here because it's pretty funny Andrej Karpathy, 75:17 ↗ (auto-captions)

Driving it by hand, and the base case

Before automating the order, he calls the closures manually, top down, on the neuron graph from P3: o._backward(), then n._backward(), then x1w1x2w2._backward(), then the two product nodes. Each call redraws the graph with one more layer of gradients filled in, and the numbers match the ones derived by hand in the previous part exactly — which is the point of doing it this way rather than jumping straight to backward().

The first attempt produces nothing, and the reason is worth internalizing. grad is initialized to 0.0 on every node, including the output. Every closure multiplies by out.grad. Multiply anything by zero and the whole graph stays zero. So the driver needs a base case: set the output's gradient to 1.0, on the grounds that the derivative of the output with respect to itself is 1. That single seed is what makes all the other numbers mean "derivative of o with respect to this node" rather than "derivative of something else."

With o.grad = 1.0 and the closures called in order, the neuron comes out like this — o.data is 0.7071, so the tanh's local derivative 1 − o² is exactly 0.5, and that 0.5 is what flows through the rest of the graph:

The zero on w2 is not a bug and is worth a beat of thought: x2 is 0, so nudging w2 cannot change the product x2*w2, and therefore cannot change the output. A dead input kills the gradient on its weight. That is a real property of trained networks, not a quirk of this toy — it is the same mechanism that makes gradients vanish behind a saturated or zeroed unit.

Topological order: why you cannot just recurse

Calling the closures by hand works only because Karpathy knows the graph and picks the order himself. The rule he is following without saying it out loud: a node may only run its _backward once its own grad is final — that is, once every node that consumes it has already deposited its contribution. Push too early and you propagate a partial number, and nothing downstream will ever come back to correct it.

The naive alternative — recurse from the output and call each child's _backward as soon as you reach it — fails on any graph that is not a simple chain. Consider a diamond: one node feeding two different consumers that later merge. Depth-first will reach the shared node through the first consumer, propagate from it while its gradient is still half-assembled, and then reach it again through the second. This is exactly the structure of every neural network, where one activation feeds many downstream units, so it is not a corner case.

The scheduling problem has a standard answer, and Karpathy explicitly declines to derive it — he points at the Wikipedia article and moves on, which is the right call for a lecture but does mean the one genuinely graph-theoretic idea in micrograd arrives as a black box. A topological order of a directed acyclic graph is a linear arrangement in which every edge points forward. The expression graph is a DAG by construction: edges run from inputs to outputs, and you cannot build a cycle by evaluating Python expressions. So a topological order exists, and running it backwards guarantees that when you reach a node, everything that depends on it has already been visited — precisely the invariant the closures need.

topo = []
visited = set()
def build_topo(v):
  if v not in visited:
    visited.add(v)
    for child in v._prev:
      build_topo(child)
    topo.append(v)
build_topo(o)

Eight lines, and the whole correctness argument sits in where topo.append(v) is placed: after the loop over children, not before. A node is only added once every one of its children is already in the list, which makes the invariant "everything to my left in this list is something I depend on" true by induction. The visited set gives each node exactly one entry no matter how many consumers it has, so the backward pass costs one closure call per node — linear in the size of the graph, not exponential in its depth. Print the list on the neuron and o is last, n second to last, the leaves first.

Two things this implementation does not handle, worth knowing before you reuse it: it is recursive, so a deep enough graph (a long unrolled loop, say) will hit Python's recursion limit, and it assumes acyclicity rather than detecting a violation. Neither matters for the lecture. Both are why real frameworks use an explicit stack.

backward(), hidden inside Value

The last manual step is calling _backward() in a loop, so that gets folded into a public method on Value. The naming convention is now doing real work: _backward with the underscore is one node's chain-rule step, backward without it is the whole graph's pass.

def backward(self):

  topo = []
  visited = set()
  def build_topo(v):
    if v not in visited:
      visited.add(v)
      for child in v._prev:
        build_topo(child)
      topo.append(v)
  build_topo(self)

  self.grad = 1.0
  for node in reversed(topo):
    node._backward()

Three moves, in this order: build the schedule starting from whatever node you called it on; seed that node's gradient at 1.0; walk the schedule in reverse, calling each node's stored step. Note self.grad = 1.0 is a plain assignment, deliberately — it is a base case, not a contribution — while everything inside the closures accumulates. And note that the topological sort starts at self, not at some global root: backward() only ever touches the subgraph that the node you called it on actually depends on. Call it on an intermediate node and you get derivatives of that node with respect to its ancestors, which is occasionally exactly what you want for debugging.

At this point o.backward() reproduces the entire table above in one call, and P3's twenty-odd cells of hand-typed chain rule are obsolete.

The bug: a node used twice

Everything so far has been tested on graphs where each value is consumed exactly once, and that hid a genuine error. The smallest case that exposes it:

a = Value(3.0, label='a')
b = a + a   ; b.label = 'b'
b.backward()

The forward pass is right: b.data is 6.0. The gradient is not. High-school calculus says the derivative of a + a with respect to a is 2. The engine reports 1.

The cause is one line down in the addition closure, and it is not the _prev set collapsing the two edges into one — that is a red herring people reliably chase. Inside __add__, self and other are bound to the same object. With plain assignment, the closure body writes a.grad = 1.0 and then immediately writes a.grad = 1.0 again to the same attribute. The second write does not add to the first; it replaces it. Two contributions of 1 arrive and one survives.

The second example makes the same failure look less like a curiosity. Take a shared input feeding two different operations that then merge:

a = Value(-2.0, label='a')
b = Value(3.0, label='b')
d = a * b    ; d.label = 'd'
e = a + b    ; e.label = 'e'
f = d * e    ; f.label = 'f'

f.backward()

Both a and b reach f along two distinct paths, through the product and through the sum. Whichever of d or e happens to run its closure last simply overwrites what the other one deposited, so the answer is not merely wrong, it depends on the iteration order of a set:

The mathematics being violated is the multivariate chain rule. When a variable influences the output through several routes, the total derivative is the sum of the derivatives along each route. Nothing in the code was expressing that sum.

the solution there is basically that we have to accumulate these gradients these gradients add Andrej Karpathy, 85:03 ↗ (auto-captions)

So every = inside every closure becomes +=. The fix is correct precisely because grad is initialized to 0.0 in the constructor: zero is the identity for addition, so a node used exactly once still ends up with exactly the value the old code gave it, while a node used n times collects all n contributions. With += in place, a + a gives 2.0 and the diamond gives −3.0 and −8.0 — the numbers the calculus predicts. It also silently makes a * a work: the closure fires twice against the same object and lands on 2·a, which is the derivative of a².

The code at the end of this part

class Value:

  def __init__(self, data, _children=(), _op='', label=''):
    self.data = data
    self.grad = 0.0
    self._backward = lambda: None
    self._prev = set(_children)
    self._op = _op
    self.label = label

  def __repr__(self):
    return f"Value(data={self.data})"

  def __add__(self, other):
    out = Value(self.data + other.data, (self, other), '+')

    def _backward():
      self.grad += 1.0 * out.grad
      other.grad += 1.0 * out.grad
    out._backward = _backward

    return out

  def __mul__(self, other):
    out = Value(self.data * other.data, (self, other), '*')

    def _backward():
      self.grad += other.data * out.grad
      other.grad += self.data * out.grad
    out._backward = _backward

    return out

  def tanh(self):
    x = self.data
    t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1)
    out = Value(t, (self, ), 'tanh')

    def _backward():
      self.grad += (1 - t**2) * out.grad
    out._backward = _backward

    return out

  def backward(self):

    topo = []
    visited = set()
    def build_topo(v):
      if v not in visited:
        visited.add(v)
        for child in v._prev:
          build_topo(child)
        topo.append(v)
    build_topo(self)

    self.grad = 1.0
    for node in reversed(topo):
      node._backward()

Line by line against the shipped library, which is almost identical: the constructor's _backward = lambda: None and _prev = set(_children) are engine.py L5–L11 — the library drops label, which only ever existed to make draw_dot readable. The addition closure is L13–L22, where the redundant 1.0 * is dropped and a coercion line (other if isinstance(other, Value) else Value(other)) is added so you can write x + 1; that coercion is built in P5. The multiplication closure is L24–L33, unchanged. backward() including build_topo is L54–L70, character-for-character the same logic. The one method here with no counterpart in the library is tanh — the shipped engine offers ReLU and pow instead, whose closures follow the identical local-derivative-times-out.grad template. That template is the whole extension mechanism: to add an operation to micrograd you write a forward line and a closure, and the scheduler needs no changes at all.

Where people get stuck

• "Why does a plus just copy the gradient?" Because ∂(a+b)/∂a = 1 and ∂(a+b)/∂b = 1 — shifting one addend by ε shifts the sum by exactly ε regardless of what the other addend is. Multiplying the upstream gradient by 1 is a no-op, so a plus node distributes its gradient unchanged to every input. The mirror-image confusion is at the times node: the local derivative there is the other operand's data, so the two inputs' gradients get each other's forward values. If you find yourself writing self.grad += self.data * out.grad, you have swapped them.
• "Why can't the closure be called the moment I reach the node?" Because the node's own grad may not be finished. In a graph where a feeds both d and e, a.grad is only complete after both d._backward() and e._backward() have run. Propagating from a after only one of them means everything upstream of a gets a fraction of the truth, permanently. Reverse topological order is the cheapest schedule that makes "my gradient is final when my turn comes" true for every node simultaneously.
• "Doesn't += double-count on ordinary graphs?" No, and the reason is the initialization. grad starts at 0.0, and each consumer of a node contributes exactly once per backward pass, so a node with one consumer gets 0 + its single contribution — identical to what assignment produced. The failure mode += does introduce is calling backward() twice on the same graph without resetting: the second pass adds to the first and you get doubled gradients. That is the same trap as forgetting zero_grad() in PyTorch, and it is the bug that bites in P7's training loop.
• "Is the _prev set the reason a + a was wrong?" No. The set does deduplicate the two edges, which affects the picture draw_dot draws and means the node is visited once by build_topo, but the gradient error happens strictly inside the closure body, where self and other are the same object and the second write clobbers the first. Change = to += and the set stays exactly as it is while the answer becomes correct.
• "Why store a function instead of a string like '+' and switch on it?" You could — that is roughly what a tape-based or graph-compiled autograd does, and it buys you the ability to inspect and optimize the backward graph. The closure form buys something else: the captured variables. t in the tanh closure, other.data in the multiply, are forward-pass intermediates that would otherwise have to be stashed and looked up. In exchange, micrograd's backward pass is opaque — you cannot print it, differentiate it again, or run it on a GPU.

Go deeper, verified

• micrograd/engine.py — Andrej Karpathy (2020) · The finished version of everything in this part: the _backward closures at L13–L52 and backward() at L54–L70, in 94 lines total. Read it after the lecture cells and the diff is startlingly small.
• micrograd_lecture_first_half_roughly.ipynb — Andrej Karpathy (2022) · The actual notebook from the video, including the by-hand _backward() calls and both bug-demonstration cells; run them yourself before and after changing += back to =.
• Autograd mechanics — PyTorch documentation · field map extra The production version of the same three ideas — a recorded DAG, a reverse traversal, and accumulation into .grad — plus what micrograd omits: graph freeing, retain_graph, and in-place-operation hazards. Read it alongside torch.Tensor.backward, which spells out that gradients are accumulated into the leaves and that a non-scalar output needs an explicit gradient argument — the generalization of self.grad = 1.0.
• CS231n: Backpropagation, Intuitions — Andrej Karpathy / Stanford CS231n · field map extra The same author's earlier written treatment, with the "add gate is a distributor, multiply gate is a switcher" framing and an explicit section on gradients adding at branches.
• Topological sorting — Wikipedia · field map extra The reference Karpathy points at rather than deriving. The DFS-with-postorder algorithm in the article is exactly build_topo; Kahn's algorithm is the iterative alternative worth knowing for deep graphs.
• Learning representations by back-propagating errors — Rumelhart, Hinton & Williams (1986) · field map extra The paper that made this algorithm famous. Two pages; the update rule and the summation over outgoing connections are the ancestors of the +=.
• Automatic Differentiation in Machine Learning: a Survey — Baydin, Pearlmutter, Radul & Siskind (2015) · field map extra Where micrograd sits in the taxonomy: reverse-mode AD, operator overloading, define-by-run. Also explains why reverse mode is the right choice when you have one output and many inputs.

Exercises

1. Break it on purpose, then prove the fixcode — Copy the class above, change all five += back to =, and run the diamond (a = Value(-2.0), b = Value(3.0), d = a*b, e = a+b, f = d*e). Then write a numerical gradient check: a function that bumps one leaf's data by h = 1e-6, rebuilds the expression, and returns (f₂ − f₁)/h. A good answer shows the check agreeing with the += version to about five decimals on both a and b (−3.0 and −8.0) and disagreeing with the = version, and states in one sentence why the assignment version's answer is order-dependent rather than merely wrong. This numeric-vs-analytic check is the same tool section 1 of the official exercise Colab asks you to build, so doing it here front-loads that section.
2. Make the topological sort iterativecode — Replace the recursive build_topo with an explicit stack so it survives deep graphs. Build a test: a chain of 10,000 additions (v = v + Value(1.0) in a loop), call backward(), and confirm the recursive version raises RecursionError while yours returns a gradient of 1.0 on the original leaf. A good answer keeps the same postorder semantics — a node is emitted only after all its children — and says how you verified that, e.g. by asserting for every node that its index in topo is greater than all of its children's.
3. Instrument the schedule — Wrap each closure so it records the node it fired on, then run backward() on the neuron from this part and on the diamond. Check three invariants: every node's _backward ran exactly once; no node fired before all of its consumers had fired; and calling backward() a second time without resetting doubles every gradient. A good answer explains the third result in terms of += plus the missing zeroing step, and predicts what that would do to a training loop.