The Value object and the expression graph

Transcript: this part, with timestamps

Part 1 ended with a definition of the derivative you could compute for any function you could call: bump an input by a tiny h, see how much the output moved, divide. That works, and for the three-input expression d = a*b + c it gave the right slopes (−3, 2, 1). It also does not scale — a neural net has thousands to billions of inputs, and re-running the forward pass once per input is hopeless. Backpropagation is the fix, and backpropagation needs something the numerical method does not: it needs to look inside the expression, at every intermediate value, and walk from the output back to the leaves applying the chain rule. Python's floats can't help with that. a*b returns -6.0, a number with no memory of a, of b, or of the fact that a multiplication happened. This part builds the object that does remember.

Outline, with timestamps

• 19:09 — The skeleton: a class that wraps one scalar, plus __repr__ so the notebook prints something readable.
• 20:21 — __add__: Python's dunder protocol, and why a + b becomes a.__add__(b) with b as other.
• 21:24 — __mul__, and a*b + c reproducing 4.0 through two chained operator calls.
• 22:26 — The missing connective tissue: results need pointers back to the values that produced them.
• 22:56 — _children as a tuple in the signature, _prev as a set on the instance — and Karpathy's shrug about why.
• 23:58 — _op: the string that says which operation made this node, empty for leaves.
• 25:00 — draw_dot pasted in: trace collects nodes and edges, graphviz renders them left to right.
• 26:02 — The fake op nodes: little circles that are not Values, inserted so the picture reads.
• 27:05 — label, and naming the intermediate e = a*b so the drawing is legible.
• 28:05 — One layer deeper: f = -2.0, L = d * f, output −8. The running example for the next twenty minutes.
• 29:05 — Recap of the forward pass, and the statement of what backprop will compute: dL/d(everything).
• 30:36 — self.grad = 0.0 added to the constructor, and rendered into the graph beside the data.

A number that remembers

The class starts as small as a class can be. Value holds one attribute, data, and one method, __repr__, which exists purely so that typing a in a notebook cell prints Value(data=2.0) instead of the default <__main__.Value object at 0x7f…>. That is not a detail worth much thought, but Karpathy calls it out around 21:55 because the entire lecture is conducted by reading printed output in a notebook — if you can't see what an object is, you can't teach with it.

The first real move is operator overloading. Python routes a + b to type(a).__add__(a, b), so defining __add__ on Value makes the + symbol mean whatever you want it to mean for these objects. Inside, the arithmetic itself is ordinary: self.data + other.data is float addition, because .data really is a Python float. The interesting part is the return value — not a float, but a new Value wrapping the result. Same for __mul__. With both defined, a*b + c works exactly as it reads: Python calls a.__mul__(b), gets back a fresh Value(-6.0), then calls .__add__(c) on that, and out comes Value(4.0) — the same 4.0 the plain-float version produced in Part 1. Nothing has been gained yet except the wrapper. The wrapper is the point: every operation now passes through code Karpathy controls, and code he controls can record things.

Two closely-related things get recorded. First, the operands. __add__ passes (self, other) into the new node's constructor as _children, and the constructor stores set(_children) as self._prev. Now d._prev is {Value(data=-6.0), Value(data=10.0)} — the intermediate a*b and the leaf c — and you can follow those pointers down until you hit nodes whose _prev is empty. Those are the leaves: values you constructed directly rather than computed. Second, the operation. A node knowing which two values made it is not enough to differentiate it; you also need to know whether they were added or multiplied, because the local derivative differs. So a second field, _op, holds a one-character string: '+', '*', or the empty string for a leaf. Between _prev and _op, any node in the graph can answer "how was I made, and from what" — which is precisely the information the chain rule needs at that node, and nothing more.

The tuple-in, set-out asymmetry is the first place where the lecture is honestly rough. The parameter is a tuple for convenience at the call site; the attribute is a set. Asked why, Karpathy checks his own old code and comes up empty:

This is how I did it in the original micrograd… I can't remember exactly the reason. I believe it was efficiency.— 22:56

Take him at his word and note the consequence, because it matters later: a set deduplicates. If you write b = a + a, the node's _prev contains one element, not two, since self and other are the same object. That does not break anything — traversal only needs to reach each distinct node once, and the per-operation gradient rule Karpathy writes in P4 accumulates into self.grad and other.grad separately even when they alias — but it is the reason a node reached by two different paths is a real hazard and a node reached twice through the same edge is not. The published library keeps the same design: self._prev = set(_children) on line 10 of engine.py, four years and one refactor later.

draw_dot: the picture the rest of the lecture runs on

At 25:00 Karpathy pastes in about twenty lines of graphviz code and explicitly declines to explain them line by line — "slightly scary code," he calls it, and moves on. That is a reasonable call: nothing in draw_dot is load-bearing for understanding backpropagation. But it is worth reading once, because its structure mirrors the structure of backward() that arrives in P4.

trace(root) is a depth-first walk that starts at the output and follows _prev pointers, collecting a set of nodes and a set of (child, parent) edges. The guard if v not in nodes makes it terminate and makes it visit each node once even in a diamond-shaped graph. That is the same recursion, with the same visited-set guard, that build_topo will use to produce a topological order in P4 — the only difference is that build_topo appends to a list after recursing into children, which is what makes the ordering come out right. If you understand trace here, you have already understood the traversal half of backprop.

The rendering half has one wrinkle worth naming. Each Value becomes a rectangular graphviz record node showing its label, data, and (after 30:36) grad. But the little circles marked + and * in the diagrams are not Values — they are extra nodes invented inside draw_dot, named uid + n._op, and wired so that the children point at the op node and the op node points at the result. Karpathy is clear about this at 26:34: only the boxes are real objects. It matters when you start reading the pictures closely, because the op circles have no data and no grad and will never get one; they are punctuation.

Labels come next, and they are pure ergonomics: a label='' keyword on the constructor, set explicitly for leaves and assigned after the fact for intermediates (e = a*b; e.label = 'e'). Karpathy calls the naming "kind of naughty" — one-letter variables assigned on the same line as the expression — and it is, but it makes the rendered graph readable, which is the whole point. Note that the label is not derived from the Python variable name; it has to be handed over by hand, which is why he forgets to set L.label the first time and gets an empty box. The finished library drops label entirely — it isn't in engine.py's constructor — because it exists only to serve the lecture's diagrams.

The running example, one layer deeper

With drawing in place, Karpathy grows the expression by one level. d is no longer the output; a new leaf f = -2.0 multiplies it to give L = d * f = -8.0. The capital L is deliberate — it stands for loss, and the whole graph is a rehearsal of the shape a real training step will have: some leaves that are weights, some leaves that are data, a scalar at the end.

The extra layer is not decoration. With d as the output, the chain rule has nothing to chain — every gradient is one local derivative deep, and you can't tell the difference between "apply the local rule" and "do the whole algorithm." With L on top, a's influence on the output has to pass through e, then d, then the final multiply. That is a three-link chain, long enough that the recursive structure of backprop is visible rather than assumed. This is the expression the next part backpropagates entirely by hand.

The state of the graph at the end of this part, all seven nodes:

Every number in the data column is the forward pass. Every number in the grad column is a placeholder. That is the honest summary of where the lecture stands at 32:10.

grad: an empty field with a precise meaning

The last two minutes of this part add a single line to the constructor — self.grad = 0.0 — and then spend the time explaining what it will mean, which is the right ratio. Three things are worth pinning down.

It is always a derivative with respect to one specific output. n.grad does not mean "the derivative of n." It means dL/dn: how much the single scalar at the root of the graph changes when n changes by a hair. Which node counts as L is a choice you make when you call backward, and it is the reason the field lives on every node while the semantics point at exactly one of them. Karpathy states this at 29:35 and it is the single most common source of confusion in the whole lecture.

Zero is a meaningful initial value, not a null. A gradient of zero says this value does not move the output at all — nudge it and nothing happens. Starting every node there means the graph begins in a state that is wrong but interpretable, and every step of backprop replaces a zero with a real number.

At initialization we're assuming that every value does not affect the output.— 31:07

That choice is not merely cosmetic. In P4 the per-operation rules write self.grad += … rather than =, precisely so that a node feeding two consumers sums the contributions from both — and summing only works if the starting point is the additive identity. And in P7, the notorious training bug is forgetting to reset these fields to zero between steps, so stale gradients from the previous iteration accumulate on top of the new ones. Both of those depend on the decision made quietly here.

Leaves are not all the same kind of leaf. At 30:36 Karpathy draws the distinction that motivates the whole exercise: in a neural net, some leaves are weights and some are input data. The gradients with respect to the weights are what you act on, because weights are what you're allowed to change. The gradients with respect to the data get computed too — the algorithm doesn't know the difference — and are simply ignored, because the dataset is fixed. Nothing in the Value class encodes this distinction; it lives entirely in what the caller decides to do with the numbers afterward.

The code at the end of this part

The saved notebook stores the Value class in its finished, end-of-lecture state, so cell 7 as checked in already contains _backward, tanh, and backward(). Reconstructed to where the video actually stands at 32:10 — no gradients computed yet, only the machinery for holding them — the class is this:

class Value:

  def __init__(self, data, _children=(), _op='', label=''):
    self.data = data
    self.grad = 0.0          # dL/d(self); 0.0 means "no effect on the output"
    self._prev = set(_children)
    self._op = _op
    self.label = label

  def __repr__(self):
    return f"Value(data={self.data})"

  def __add__(self, other):
    out = Value(self.data + other.data, (self, other), '+')
    return out

  def __mul__(self, other):
    out = Value(self.data * other.data, (self, other), '*')
    return out


a = Value(2.0, label='a')
b = Value(-3.0, label='b')
c = Value(10.0, label='c')
e = a*b; e.label = 'e'
d = e + c; d.label = 'd'
f = Value(-2.0, label='f')
L = d * f; L.label = 'L'
L                              # Value(data=-8.0)

And the visualizer, verbatim from cell 8 of the notebook, with the grad field already in the node label as it is by 31:38:

from graphviz import Digraph

def trace(root):
  # builds a set of all nodes and edges in a graph
  nodes, edges = set(), set()
  def build(v):
    if v not in nodes:
      nodes.add(v)
      for child in v._prev:
        edges.add((child, v))
        build(child)
  build(root)
  return nodes, edges

def draw_dot(root):
  dot = Digraph(format='svg', graph_attr={'rankdir': 'LR'}) # LR = left to right

  nodes, edges = trace(root)
  for n in nodes:
    uid = str(id(n))
    # for any value in the graph, create a rectangular ('record') node for it
    dot.node(name = uid, label = "{ %s | data %.4f | grad %.4f }" % (n.label, n.data, n.grad), shape='record')
    if n._op:
      # if this value is a result of some operation, create an op node for it
      dot.node(name = uid + n._op, label = n._op)
      # and connect this node to it
      dot.edge(uid + n._op, uid)

  for n1, n2 in edges:
    # connect n1 to the op node of n2
    dot.edge(str(id(n1)), str(id(n2)) + n2._op)

  return dot

draw_dot(L)

Line by line, the parts that repay attention:

• self._prev = set(_children) — the backward-pointing edges. Survives verbatim into the library at engine.py line 10; the lecture's label parameter does not.
• out = Value(self.data + other.data, (self, other), '+') — three jobs in one expression: compute, wire, tag. Compare engine.py lines 13–22, which are identical apart from a coercion line for raw numbers and the _backward closure added in P4.
• grad = 0.0 in the constructor rather than in the operators — leaves get one too, and leaves are exactly the nodes whose gradients you'll act on.
• if v not in nodes inside build — the visited guard. Without it a diamond-shaped graph is walked exponentially, and a repeated subexpression is drawn twice.
• uid = str(id(n)) — graphviz needs string names, and object identity is the only thing guaranteed unique. Two distinct Values holding 4.0 are still two nodes; equality is never consulted anywhere in micrograd, which is also why _prev can be a set of unhashable-looking objects (default identity hashing).
• rankdir: 'LR' and the edge direction child → op → parent — the arrows point the way the forward pass flows. Backprop runs against the arrows, which is worth holding in your head when you stare at these diagrams in P3.

Where people get stuck

• "Why does Value(2.0) + 1 blow up?" Because the lecture's __add__ reaches straight for other.data and a Python int has no .data. There is no coercion yet, and no __radd__ either, so 1 + Value(2.0) fails differently. Both gaps are real and both are filled later — coercion in P5, and permanently in the library's line 14: other = other if isinstance(other, Value) else Value(other). If you're typing along, wrap your constants.
• "Are the + and * circles nodes in the graph?" No. They exist only inside draw_dot, are named by string concatenation, and hold no state. The graph trace returns contains rectangles only. Nothing in backprop ever touches an op circle — the operation is a string field on the result node, and even that string is never read by the algorithm, only by the visualizer and by you.
• "grad is 0.0 everywhere — is it broken?" No; nothing has computed anything yet. At 32:10 the class can only hold gradients, not produce them. The distinction between "the field exists and is initialized" and "the field has been filled by a backward pass" is exactly the boundary between this part and the next one, and it is why Karpathy spends two minutes on a one-line change.
• "Why store the operands at all — can't you just re-run the expression?" You can, and that is what the numerical method in P1 does: it re-runs the whole forward pass once per input. The graph exists so you don't have to. Storing _prev costs one set per node and buys you every gradient in a single backward sweep instead of one forward pass per parameter. For a network with a million weights that is a factor-of-a-million difference, and it is the only reason deep learning is computationally possible.

Go deeper, verified

• micrograd_lecture_first_half_roughly.ipynb — Andrej Karpathy (2022) · cells 7–9 are this part; note the saved class is the end-of-lecture version, so read it knowing _backward/tanh/backward arrive in P4–P5.
• micrograd/engine.py — Andrej Karpathy · the finished 94-line class; lines 5–11 are this part's constructor and lines 13–33 this part's two operators, grown up.
• Emulating numeric types — Python data model — Python Software Foundation · field map extra: the full dunder table, including the reflected __radd__/__rmul__ that micrograd needs before 2 * neuron_output works.
• Autograd mechanics — PyTorch docs · field map extra: the same idea at production scale — a DAG recorded during the forward pass, leaves, and why the graph is rebuilt every iteration rather than reused.
• Backpropagation, intuitions (CS231n) — Andrej Karpathy et al., Stanford · field map extra: Karpathy's own written treatment of the circuit-diagram view of an expression, with the same forward-then-backward pass over a small graph.
• Graphviz node shapes — Graphviz · field map extra: what the record shape and the { a | b | c } label syntax in draw_dot actually mean, if you want to change what the boxes display.
• Learning representations by back-propagating errors — Rumelhart, Hinton & Williams, Nature (1986) · field map extra: the canonical reference for the algorithm this data structure exists to support. Read it after P4, not before.

Exercises

1. Type it, don't paste itcode — In an empty notebook, write Value from memory: data, grad, _prev, _op, label, __repr__, __add__, __mul__. Then build the running example and check four things: L.data == -8.0; d._op == '+'; len(a._prev) == 0; and that {v.label for v in L._prev} == {'d', 'f'}. A good answer gets all four on the first run and can say, without looking, which of the seven nodes have an empty _op and why.
2. A tracer with no graphvizcode — draw_dot needs a system graphviz install that not every environment has. Write show(root, indent=0) that prints the graph as an indented tree — one line per node with label, data, grad, and op, recursing into _prev — and run it on L. Then extend it with a visited set and explain in a comment what changes for the expression x = a*b; y = x + x. A good answer notices that the indented form duplicates shared subexpressions while the real graph does not, and that y._prev has one element rather than two.
3. Predict the picture, then draw it — Before running anything, sketch on paper the graph for L2 = (a*b + c) * (a + c) with the same leaf values: how many rectangles, how many op circles, how many arrows, and what L2.data is. Then build it and check against draw_dot. A good answer gets 7 rectangles (3 leaves plus 4 results — a and c are each reused, not duplicated), 4 op circles, 12 arrows, and 48.0; and it notices that a now reaches the output along two distinct paths, which is the exact situation that breaks a naive backward pass in P4. The official exercise Colab has no section for this part — its three sections land on P1, P5 and P7 — so this stands in for it.