Autograd: What Did PyTorch Record, and Where Does the Gradient Stop?
Here is a training loop with two parameters. It runs, it does not warn, and the loss goes to zero.
import torch
x = torch.tensor([1.0, 2.0, 3.0, 4.0])
target = torch.tensor([3.0, 6.0, 9.0, 12.0])
w1 = torch.tensor(0.5, requires_grad=True)
w2 = torch.tensor(1.0, requires_grad=True)
params = [w1, w2]
feature_log = []
def stage_one(t):
f = w1 * t
detached = f.detach() # keep the values for later inspection
feature_log.append(detached)
return detached
learning_rate = 0.02
for step in range(400):
f = stage_one(x)
prediction = w2 * f
loss = ((prediction - target) ** 2).mean()
loss.backward()
with torch.no_grad():
for p in params:
if p.grad is not None:
p -= learning_rate * p.grad
p.grad.zero_()
if step in (0, 9, 99, 399):
print(f"step={step:3d} loss={loss.item():.6f} "
f"w1={w1.item():.4f} w2={w2.item():.4f}")
step= 0 loss=46.875000 w1=0.5000 w2=1.3750
step= 9 loss=11.521060 w1=0.5000 w2=3.7071
step= 99 loss=0.000009 w1=0.5000 w2=5.9979
step=399 loss=0.000000 w1=0.5000 w2=6.0000
Read the columns the way Chapter 1 taught. The loss falls to zero, which is the outcome we asked for. And w1 never moves. Not slowly, not by a little: it holds 0.5000 for four hundred steps while w2 climbs to exactly 6.0, which is the value that makes w1 * w2 equal to the 3 we were trying to learn. Half the model was frozen, the other half absorbed the entire job, and the objective was satisfied anyway.
Everything Chapters 1 and 2 gave us reports that this program is fine. The three phases are all present and all executing: a forward pass runs, backward() runs, an update runs. Every shape is what it should be, no broadcast went anywhere unexpected, and no dtype or device is out of place. There is no exception to read and no traceback to mislead us.
The evidence that something is wrong is in a place we have not yet learned to look:
print("w1.grad:", w1.grad)
print("w2.grad:", w2.grad)
w1.grad: None
w2.grad: tensor(0.)
w2 has a gradient. w1 has no gradient at all, and never did, which is why the update loop skipped it four hundred times without complaint. So backward() was called on a loss that w1 helped compute, and w1 received nothing.
That is the subject of this chapter. While all those tensor operations were executing, PyTorch was building a second thing alongside the values: a record of which operations produced which results, sufficient to differentiate the whole computation afterwards. loss.backward() traverses that record. When a gradient is missing, stale, or arrives somewhere you did not intend, the explanation is almost always in the record’s structure rather than in the arithmetic.
The question this chapter answers is:
What exactly does PyTorch remember during the forward pass, how does that become a gradient during the backward pass, and where can the path from a parameter to the loss stop existing without anything failing?
By the end you should be able to look at an unfamiliar computation and say which tensors participate in autograd, where the connections are, where .grad will be populated and where it will not, and — when a gradient is missing — find the exact operation after which the parameter stopped being connected to the loss.
We will come back to this loop and repair it with evidence rather than by guessing.
What the forward pass writes down
Start smaller than any real model, with a composition short enough to differentiate on paper.
w = torch.tensor(2.0, requires_grad=True)
x = w * 3
y = x ** 2
Three tensors, two operations. Before running anything, work out what the derivative should be. Substituting, y = (3w)² = 9w², so dy/dw = 18w, which at w = 2 is 36. Hold that number; we are going to check PyTorch against it.
Now look at what the three tensors carry. Chapter 2 had a describe helper for shape, dtype, device and strides. This chapter needs a different set of properties, so here is its counterpart:
def describe(name, t):
print(f"{name:<6} value={t.item():<8.3f} "
f"requires_grad={str(t.requires_grad):<5} "
f"is_leaf={str(t.is_leaf):<5} "
f"grad_fn={type(t.grad_fn).__name__ if t.grad_fn is not None else 'None'}")
describe("w", w)
describe("x", x)
describe("y", y)
w value=2.000 requires_grad=True is_leaf=True grad_fn=None
x value=6.000 requires_grad=True is_leaf=False grad_fn=MulBackward0
y value=36.000 requires_grad=True is_leaf=False grad_fn=PowBackward0
Three properties, and each one answers a different question.
requires_grad answers whether reverse-mode autograd needs to track a tensor for gradient computation. We set it explicitly on w. In ordinary grad mode, a differentiable operation whose result depends on at least one tensor requiring gradients will normally produce a result that also requires gradients.
That propagation is not unconditional. Gradient modes such as torch.no_grad() and torch.inference_mode() can suppress recording, and some operations produce outputs that are not differentiable. The useful rule is therefore: follow requires_grad as evidence of the recorded differentiable path, rather than assuming it propagates through every operation.
is_leaf needs a slightly more precise rule. For tensors with requires_grad=True, a leaf is one that was not produced by an operation recorded by autograd. w is therefore a leaf, while the computed tensors x and y are not.
Tensors with requires_grad=False are considered leaves by convention even when they are the result of ordinary tensor operations. So is_leaf is not simply a synonym for “created rather than computed,” and it is not a synonym for “trainable.” Its most useful role here is telling us whether autograd normally accumulates a requested gradient into this tensor’s .grad.
grad_fn answers which operation produced this value, and how do we differentiate it? MulBackward0 is the object that knows how to turn a gradient with respect to x into a gradient with respect to the inputs of the multiplication. w has none, because nothing recorded produced it.
Those three properties, in that order, are the first thing to print when autograd surprises you. The rest of this chapter is largely about learning to read them.
The record is a real object you can walk
grad_fn is not a label. It is a node in a graph, and each node knows its predecessors through next_functions:
print(y.grad_fn)
print(y.grad_fn.next_functions)
print(y.grad_fn.next_functions[0][0].next_functions)
<PowBackward0 object at 0x7f3c4b438130>
((<MulBackward0 object at 0x7f3c4b438040>, 0),)
((<AccumulateGrad object at 0x7f3b924e6140>, 0), (None, 0))
Read that from the top. The node that produced y is a power. Its one predecessor is the multiplication that produced x. That multiplication has two predecessors, because multiplication has two operands: one is an AccumulateGrad, and the other is None. The None is the constant 3, which does not require gradients and therefore has nowhere to send one.
AccumulateGrad is the interesting one, and it is worth walking the structure to see where it sits. A short recursive printer is enough:
def show_graph(fn, depth=0):
if fn is None:
return
indent = " " * depth
name = type(fn).__name__
if name == "AccumulateGrad":
print(f"{indent}{name} -> leaf tensor, value {fn.variable.item()}")
else:
print(f"{indent}{name}")
for nxt, _ in fn.next_functions:
show_graph(nxt, depth + 1)
show_graph(y.grad_fn)
PowBackward0
MulBackward0
AccumulateGrad -> leaf tensor, value 2.0
That is the record, printed. It has exactly the shape of the computation we wrote, reversed: square, then multiply, then a node whose job is to add the arriving gradient into a leaf tensor’s .grad. And it really is our leaf tensor, not a copy of it:
accumulate = y.grad_fn.next_functions[0][0].next_functions[0][0]
print("AccumulateGrad.variable is w:", accumulate.variable is w)
AccumulateGrad.variable is w: True
That single line is worth more than any metaphor about graphs. For an ordinary leaf tensor that requires gradients, AccumulateGrad is the node through which autograd accumulates the arriving gradient into .grad.
That does not mean every populated .grad must correspond to an AccumulateGrad node. A non-leaf tensor can also be told to retain its gradient with retain_grad(). The distinction is that leaf-gradient accumulation happens by default, while retaining an intermediate gradient is something we request explicitly.
You will not write this walker often. Writing it once, on a computation whose structure you already know, is what turns “PyTorch builds a graph” from a claim into something you have seen.
What backward() does with it
Now run the backward pass and compare with the 36 we derived on paper:
y.backward()
print("w.grad:", w.grad.item())
w.grad: 36.0
Three things happened, and it is worth separating them because different failures affect different ones.
A starting gradient was chosen. Backpropagation needs a seed: the derivative of the output with respect to itself. For a scalar output that is 1.0, and PyTorch supplies it. For a non-scalar output there is no single obvious seed, so it declines:
v = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
z = v ** 2
z.backward()
RuntimeError: grad can be implicitly created only for scalar outputs
You can supply the seed yourself, and doing so is occasionally the clearest way to answer a question:
z.backward(torch.ones_like(z))
print(v.grad)
tensor([2., 4., 6.])
The argument to backward() is the vector used in the Jacobian-vector product, which means it can also serve as an investigative tool. Seeding a single element with 1.0 and the rest with 0.0 isolates the contribution from that output element.
Most standard training loops reduce their objective to a scalar before calling backward(), so the seed can be inferred automatically. When the output is not scalar, inspect its shape and decide explicitly what vector-Jacobian product you are asking PyTorch to compute.
The nodes were visited in order. Starting from loss.grad_fn, autograd walks the record, and at each node applies that operation’s local derivative to the gradient arriving from above. This is the chain rule, executed one operation at a time. For our computation:
dy/dy = 1
dy/dx = 2x = 12 (PowBackward0)
dy/dw = 12 × 3 = 36 (MulBackward0, whose local derivative is 3)
Nothing like a single symbolic expression for dy/dw needs to be constructed. Each backward node implements the local derivative rule for its operation, and autograd composes those local results numerically as it traverses the graph.
The individual operations in a transformer are obviously much larger and more expensive than the scalar operations here, but the differentiation principle is unchanged: local backward rules compose into a gradient for the complete computation.
The result was accumulated into leaves. The traversal ends at AccumulateGrad nodes, and each one adds the arriving gradient into the corresponding tensor’s .grad. Chapter 1 observed that behaviour from the outside, as gradients that grew every iteration until zero_() was called. Now you can see the node responsible for it.
Why gradient contributions add
Chapter 1 justified accumulation across repeated backward() calls. There is a second reason gradients add, it is more fundamental, and it comes from the same node.
Give w two different routes to the loss:
w = torch.tensor(2.0, requires_grad=True)
a = 3 * w
b = w ** 2
loss = a + b
show_graph(loss.grad_fn)
AddBackward0
MulBackward0
AccumulateGrad -> leaf, value 2.0
PowBackward0
AccumulateGrad -> leaf, value 2.0
Two paths leave the addition and both arrive at a leaf. Differentiate it by hand: loss = 3w + w², so dloss/dw = 3 + 2w, which at w = 2 is 7.
loss.backward()
print("w.grad:", w.grad.item())
w.grad: 7.0
The two paths contributed 3 and 4, and the total is their sum. That is not a special rule PyTorch applies to branches; it is what the chain rule says when a variable influences a result through more than one route. And the mechanism is visible in the printout above, if you check whether those two AccumulateGrad lines are two nodes or one:
left = loss.grad_fn.next_functions[0][0].next_functions[0][0]
right = loss.grad_fn.next_functions[1][0].next_functions[0][0]
print("same AccumulateGrad node:", left is right)
same AccumulateGrad node: True
One node, reached twice. Each traversal adds its contribution to the same .grad.
This unifies two things that look unrelated when you meet them separately. A gradient summed over the branches of a residual connection, a gradient summed over every position where a shared embedding was used, a gradient summed over two backward calls you forgot to clear between — all three are the same node doing the same addition. Which is also why zero_grad() matters and why it is your responsibility: PyTorch cannot distinguish “these two contributions belong to the same step” from “the previous step’s gradient was never cleared”.
Broadcasting, from Chapter 2, is a special case of this worth recognizing. When a (1, 128) operand is broadcast against a (32, 128) one, the same values participate in 32 rows of the result, so the backward pass sums 32 contributions back down into the original shape. When broadcasting expanded an operand along one or more dimensions, the backward pass reduces the corresponding gradient back to the operand’s original shape by summing over those expanded dimensions. Any averaging comes from some other operation in the computation, such as a later .mean().
Computed is not the same as retained
Return to the first example and ask for the gradient of the intermediate:
w = torch.tensor(2.0, requires_grad=True)
x = w * 3
y = x ** 2
y.backward()
print("w.grad:", w.grad)
print("x.grad:", x.grad)
w.grad: tensor(36.)
x.grad: None
UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being
accessed. Its .grad attribute won't be populated during autograd.backward(). If you
indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad()
on the non-leaf Tensor.
This produces a specific and common wrong conclusion, which goes: x.requires_grad is True, x is plainly part of the computation, backward() ran, and yet x.grad is None, therefore something about the backward pass failed.
Nothing failed. The derivative dy/dx was computed — it had to be, because it is the value PowBackward0 handed to MulBackward0, and without it w.grad could not exist. It was computed, used, and discarded. There is no AccumulateGrad node behind x, so there was nowhere for it to be stored.
Hold the distinction precisely, because a great deal of confusion collapses onto it:
A gradient being computed during backpropagation and a gradient being retained in a tensor’s
.gradfield are different events. Every tensor on the path contributes to the first. By default, only leaf tensors that require gradients participate in the second.
If you need to see an intermediate gradient, say so:
w = torch.tensor(2.0, requires_grad=True)
x = w * 3
x.retain_grad()
y = x ** 2
y.backward()
print("x.grad:", x.grad.item(), " w.grad:", w.grad.item())
x.grad: 12.0 w.grad: 36.0
12 is 2x at x = 6, which is the number the chain-rule trace above predicted. retain_grad() did not change what was computed. It attached storage to a value that was previously passing through.
That is the right way to think about it, and it also tells you how to use it. retain_grad() is an instrument, not a setting. The useful question is never “should I retain gradients here” but:
Which derivative do I need to observe in order to distinguish the explanations I am currently considering?
Then retain exactly that one. Retaining intermediate gradients throughout a large model costs memory and gives you a wall of numbers rather than an answer.
Two alternatives are worth knowing, because they answer the same kind of question with less disturbance to the program.
torch.autograd.grad() returns gradients instead of accumulating them, which means it can ask a question without touching any .grad field:
w = torch.tensor(2.0, requires_grad=True)
a = w * 3
b = a ** 2
print(torch.autograd.grad(b, a, retain_graph=True)[0].item())
print(torch.autograd.grad(b, w)[0].item())
print("w.grad is still:", w.grad)
12.0
36.0
w.grad is still: None
That is the tool to reach for when you are investigating a live training loop and do not want your instrumentation to change the gradients the optimizer is about to consume.
A hook observes a gradient as it passes, without storing it:
w = torch.tensor(2.0, requires_grad=True)
u = w * 3
u.register_hook(lambda g: print("gradient flowing into u:", g.item()))
(u ** 2).backward()
gradient flowing into u: 12.0
Hooks earn their place when the interesting event is rare — a value that becomes non-finite at step 700, a gradient that is fine for most batches and enormous for one — because the hook can test a condition and stay silent otherwise. Chapter 11 uses them that way across a whole model. Here the point is narrower: the gradient exists at that boundary whether or not anyone stores it, and a hook is proof of that.
Where the path stops
Everything so far has been about a record that is intact. The interesting failures are the ones where it is not, and there are only a few mechanisms by which a connection fails to exist. Knowing them by their evidence is most of what makes missing gradients tractable.
The tensor never entered
The simplest case. A tensor created without requires_grad=True has no AccumulateGrad node, so operations on it are not recorded unless some other operand brings autograd in:
x = torch.tensor([2.0])
y = x * 3
loss = y.sum()
loss.backward()
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
This one is loud, because with no participating tensor anywhere the loss has no grad_fn at all and backward() has nothing to start from. Chapter 1 met the same message from a different cause — an update that rebound w to a fresh non-tracking tensor — which is a useful reminder that the message identifies a state, not a cause.
The important thing here is what it takes for the message not to appear. If any operand on the way to the loss requires gradients, loss.requires_grad will be True and backward() will run happily, whatever happened to the other operands. A loss that requires gradients tells you that something is connected. It does not tell you that the thing you care about is.
The path was cut: detach()
detach() returns a tensor that holds the same values as the original and is disconnected from the history that produced them:
w = torch.tensor([2.0, 3.0], requires_grad=True)
h = w * 3
d = h.detach()
describe_flags = lambda n, t: print(f"{n:<3} requires_grad={t.requires_grad} "
f"is_leaf={t.is_leaf} grad_fn={t.grad_fn}")
describe_flags("h", h)
describe_flags("d", d)
print("same storage:", d.data_ptr() == h.data_ptr())
h requires_grad=True is_leaf=False grad_fn=<MulBackward0 object at ...>
d requires_grad=False is_leaf=True grad_fn=None
same storage: True
Note the last line. detach() did not copy anything; the two tensors share the same values in the same memory. What changed is the record: d has no grad_fn, so nothing downstream of d can lead back to w. It is a leaf again, in a computation that started at the point of detachment.
That sharing has a consequence you should know about before it surprises you. Writing through the detached tensor modifies the original:
with torch.no_grad():
d[0] = 99.0
print("h:", h)
h: tensor([99., 9.], grad_fn=<MulBackward0>)
If h had been saved by some node for its backward pass, that write would produce an error later, at the backward call rather than here. We come back to that mechanism at the end of the chapter.
The reason detach() deserves this much attention is not that it is dangerous. It is a legitimate and frequently necessary operation: it is useful when logging or storing values without retaining their upstream graph, when deliberately truncating a gradient path, or when constructing a target that should not propagate gradients into the computation that produced it.
For freezing model parameters themselves, the more direct mechanism is normally to set those parameters’ requires_grad flags to False. A detach is better thought of as a boundary in a particular computation than as the general mechanism for freezing a module. The reason it deserves attention is that it fails quietly. Code downstream of a detach continues to produce entirely sensible numbers. Nothing raises. The only symptom is that gradients stop arriving somewhere upstream, and, as the opening loop showed, the loss may go down anyway.
So the question to attach to every detach you meet in unfamiliar code is:
Was this boundary intended, and does it sit where the author thought it did?
The region was not recorded: torch.no_grad()
torch.no_grad() looks like it does the same thing, and describing them both as “turning off gradients” is the source of a lot of imprecise debugging. They operate on different objects.
detach() acts on one tensor, cutting its link to prior reverse-mode autograd history. torch.no_grad() acts on a region of code: ordinary differentiable tensor operations executed inside that region are excluded from the reverse-mode autograd graph.
There is one implementation detail worth knowing if you are debugging unusual code: tensor factory functions that explicitly accept a requires_grad argument are an exception to no_grad, and no_grad does not disable forward-mode automatic differentiation. The difference shows up as soon as a second operand is involved:
p = torch.tensor(1.0, requires_grad=True)
q = torch.tensor(2.0, requires_grad=True)
out1 = p.detach() * q
with torch.no_grad():
out2 = p * q
print("out1:", out1.requires_grad, type(out1.grad_fn).__name__)
print("out2:", out2.requires_grad, out2.grad_fn)
out1: True MulBackward0
out2: False None
Detaching p removed one operand from the record, and the multiplication was still recorded because q remained. Gradients flow to q and not to p. Inside no_grad, the multiplication itself was not recorded, and neither operand receives anything.
Two further properties are worth stating, because both get assumed incorrectly.
no_grad does not modify the tensors it touches. p.requires_grad is still True afterwards, and operations on p outside the block are recorded normally. It is a property of the block, not a property that leaks onto the values that passed through it. This is exactly why Chapter 1’s parameter update works: the arithmetic is unrecorded, and w remains a trainable leaf on the other side.
And the current state is directly observable, which matters when you suspect that a framework, a callback or an evaluation wrapper has put you inside a block you did not write:
print(torch.is_grad_enabled())
There is also torch.enable_grad(), which re-enables recording inside a no_grad region, and torch.inference_mode(), which is more aggressive than no_grad and produces tensors carrying a restriction with them:
w = torch.tensor(2.0, requires_grad=True)
with torch.inference_mode():
a = w * 3
b = a * w
b.backward()
RuntimeError: Inference tensors cannot be saved for backward. Please do not use
Tensors created in inference mode in computation tracked by autograd.
That message is the reason to know inference_mode exists at this point in the book. Inference mode is designed for computations that will not interact with autograd later. By disabling additional autograd bookkeeping beyond no_grad, it can provide further performance benefits. The trade-off is stricter: tensors created in inference mode cannot subsequently participate in computations that autograd needs to record. The failure surfaces wherever that reuse happens, which may be a long way from the block that produced the tensor. no_grad has no such restriction, and is the safer choice whenever a value computed without gradients might be used inside one later.
The value left the tensor system
.item() extracts a scalar tensor’s value into a Python number. NumPy conversion creates the same kind of boundary, but a tensor that requires gradients cannot normally be converted with a plain .numpy() call. In practice you will usually see something such as t.detach().cpu().numpy() when the value is being exported for observation.
Once a value has left the autograd-connected tensor computation, rebuilding a new tensor from that value does not reconstruct the history that produced it:
w = torch.tensor(2.0, requires_grad=True)
prediction = w * 3
bad = torch.tensor((prediction - 6.0).item() ** 2, requires_grad=True)
bad.backward()
print("bad.grad:", bad.grad, " w.grad:", w.grad)
bad.grad: tensor(1.) w.grad: None
Look at what that produced. backward() ran without complaint. A gradient exists. It is the gradient of bad with respect to itself, in a graph containing exactly one tensor, and it tells you nothing about w. The requires_grad=True in that constructor did not reconnect anything; it created a brand new starting point.
This is the clearest small example of a principle that will recur:
requires_grad=Truedeclares that gradients should be tracked from this tensor forward. It cannot restore a connection to a computation that happened earlier.
The same applies to a round trip through NumPy. If the operation exists in PyTorch, keep it in PyTorch; use .item() and .numpy() for observation, not for computation that needs to be differentiated.
Find the first broken edge
Chapter 2’s central technique was to find the first wrong tensor rather than the first illegal operation, because the line that raises is frequently downstream of the line that was mistaken. Gradients need the same move, adapted:
Find where the path from the parameter to the loss stops existing, rather than the line where
.gradhappened to beNone.
The reason this matters is that the two lines are almost never the same. w1.grad is None is discovered wherever you happen to inspect it — in the optimizer step, in a logging line, at the end of training. The break is wherever the record was cut. Everything between those two points is intact, and changing anything in that region cannot help.
There are two directions to search, and they suit different situations.
Forwards, from the parameter
Instrument the forward pass and watch the three properties change. This is the version to use when you can run the computation and you have the source in front of you:
def track(name, t):
print(f"{name:<14} requires_grad={str(t.requires_grad):<5} "
f"is_leaf={str(t.is_leaf):<5} "
f"grad_fn={type(t.grad_fn).__name__ if t.grad_fn is not None else 'None'}")
return t
Applied to the opening loop, one step, written out flat so every intermediate is visible:
track("w1", w1)
raw = track("w1 * x", w1 * x)
f = track("stage_one out", raw.detach())
pred = track("w2 * f", w2 * f)
err = track("pred - target", pred - target)
loss = track("loss", (err ** 2).mean())
w1 requires_grad=True is_leaf=True grad_fn=None
w1 * x requires_grad=True is_leaf=False grad_fn=MulBackward0
stage_one out requires_grad=False is_leaf=True grad_fn=None
w2 * f requires_grad=True is_leaf=False grad_fn=MulBackward0
pred - target requires_grad=True is_leaf=False grad_fn=SubBackward0
loss requires_grad=True is_leaf=False grad_fn=MeanBackward0
The trace names the boundary exactly. requires_grad is True, then False, then True again, and the transition happens at a single line. The tensor at that line is a leaf with no grad_fn, which is the signature of a value that entered the computation from outside rather than being computed within it.
The last three rows are the part worth studying, because they are what makes this bug survive. loss.requires_grad is True. loss.grad_fn exists. backward() will run and populate a gradient. All of that is a statement about w2, and none of it is a statement about w1. Checking whether the loss is connected to the graph is a real check, and it is the check that this class of bug passes.
Backwards, from the loss
The other direction asks the graph itself which leaves it can reach. This is the version to use when the forward pass is spread across modules and helper functions and you cannot conveniently instrument every line — which describes most real code.
def reachable_leaves(loss):
"""Every leaf tensor that backward() from this loss would accumulate into."""
found, seen, stack = [], set(), [loss.grad_fn]
while stack:
fn = stack.pop()
if fn is None or id(fn) in seen:
continue
seen.add(id(fn))
if type(fn).__name__ == "AccumulateGrad":
found.append(fn.variable)
for nxt, _ in fn.next_functions:
stack.append(nxt)
return found
f = (w1 * x).detach()
loss = ((w2 * f - target) ** 2).mean()
reached = {id(t) for t in reachable_leaves(loss)}
for name, p in (("w1", w1), ("w2", w2)):
print(f"{name}: reachable from loss = {id(p) in reached}")
w1: reachable from loss = False
w2: reachable from loss = True
That answers the reachability question before backward() has been called. Conceptually, it walks the same predecessor structure that the autograd engine will later use, but this helper is not a reimplementation of the autograd engine.
It also deliberately reaches below the level of API you should normally build application logic on: concrete node names such as AccumulateGrad and attributes such as .variable are useful here for understanding and debugging, but production code should prefer supported interfaces such as torch.autograd.grad() when possible.
There is a lighter way to ask about a single tensor, using the tool from earlier:
g = torch.autograd.grad(loss, [w1, w2], allow_unused=True, retain_graph=True)
print("d loss/d w1:", g[0])
print("d loss/d w2:", g[1])
d loss/d w1: None
d loss/d w2: tensor(-18.7500)
allow_unused=True is what makes this a diagnostic rather than an error. Without it, PyTorch raises The differentiated Tensor at index 0 appears to not have been used in the graph, which is the same finding delivered as an exception. Either form answers a precise question: is there a differentiable path from this loss to this tensor, yes or no.
Repair, then prove the repair
The break is at the return statement of stage_one, which returns the detached copy that was made for the log rather than the tensor that was computed:
def stage_one(t):
f = w1 * t
feature_log.append(f.detach())
return f
Now the verification, and this is the part that distinguishes a fix from a coincidence. The loss went to zero before the repair, so “the loss goes down” proves nothing here. What we claimed was broken was specifically the path from w1 to the loss, so the evidence has to be about w1:
step= 0 loss=46.875000 w1=1.2500 w2=1.3750
step= 9 loss=0.031576 w1=1.7079 w2=1.7861
step= 99 loss=0.000000 w1=1.6931 w2=1.7719
step=399 loss=0.000000 w1=1.6931 w2=1.7719
print("w1 * w2 =", (w1 * w2).item())
w1 * w2 = 3.0000007152557373
w1 moved, which it never did before. Both parameters now receive gradients, and their product converges to the multiplier we were trying to learn. That is the evidence the diagnosis predicted.
State the general form of this, because it applies to every repair in this book:
Name the quantity that was wrong, say what it should become, and observe that it now does. A symptom disappearing is not the same as a mechanism being restored.
Why is .grad None?
You now have the pieces to answer this properly rather than by trying things. The question decomposes, and each part has evidence attached.
Did backward() actually run? A fresh leaf tensor that requires gradients normally begins with .grad is None. After that, the field may contain an accumulated gradient, may have been explicitly zeroed, or may have been reset to None, so interpret its current value in the context of the training loop. This sounds too obvious to check until you meet code where an exception, an early continue, a gradient-accumulation schedule or a conditional branch skipped the call for this particular step.
Does the tensor require gradients? t.requires_grad. If it is False, no gradient will be stored regardless of anything else. But before setting it to True, ask why it is False. A parameter that was deliberately frozen, a tensor that was replaced by a detached copy and a tensor that was created without the flag are three different situations with three different repairs, and only one of them is fixed by requires_grad_(True).
Is it a leaf? t.is_leaf. If it requires gradients and is not a leaf, then .grad is None is the documented default and nothing has gone wrong. The derivative was computed and used during the backward pass; there was simply nowhere to store it. Use retain_grad() or torch.autograd.grad() if you need to see it.
Is it connected to this loss? The reachability check, or torch.autograd.grad(loss, t, allow_unused=True). This is the question that the previous three cannot answer, and it is the one that catches the silent failures. A tensor can require gradients, be a leaf, and be entirely absent from the graph the loss was built on.
If it is disconnected, where? Trace forwards from the tensor and find the operation after which requires_grad becomes False, or walk backwards from the loss and find where the branch you expected is missing. The candidates are the mechanisms above: a detach(), a no_grad region, a value extracted to Python or NumPy and rebuilt, or a tensor that never required gradients in the first place.
Am I inspecting the object that participated? Python names can be rebound. If self.weight was reassigned, or a parameter was replaced rather than modified in place, the tensor in the forward pass and the tensor you are printing may be different objects — the one that received the gradient and the one you are looking at. id() before and after settles it. Chapter 1 met this with a rebinding update, and Chapters 4 and 5 return to it once optimizers and modules hold their own references.
One distinction is worth pulling out because the two states look similar and mean opposite things:
w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(2.0, requires_grad=True)
x = torch.tensor(0.0)
loss = (w * x + b - 1.0) ** 2
loss.backward()
print("w.grad:", w.grad, " b.grad:", b.grad)
w.grad: tensor(0.) b.grad: tensor(2.)
w.grad is tensor(0.), not None. w is connected, the gradient was computed, and it is genuinely zero because the input it multiplies is zero. After a known backward pass on a leaf tensor that requires gradients, an explicit zero tells you that the parameter was connected and the derivative evaluated to zero. None requires more investigation: the tensor may have been disconnected, backward may not have reached it, the field may have been reset to None, or you may be inspecting a non-leaf whose gradient was not retained.
So zero and None are different diagnostic states, but neither should be interpreted without knowing what happened immediately before you inspected them. Confusing them sends you to look for a graph break where the real question is why an input is zero, or the reverse.
The graph has a lifetime
Call backward() twice on the same loss and PyTorch objects:
w = torch.tensor(2.0, requires_grad=True)
loss = (w * 3) ** 2
loss.backward()
loss.backward()
RuntimeError: Trying to backward through the graph a second time (or directly access
saved tensors after they have already been freed). Saved intermediate values of the
graph are freed when you call .backward() or autograd.grad(). Specify
retain_graph=True if you need to backward through the graph a second time or if you
need to access saved tensors after calling backward.
The standard response is to add retain_graph=True, and the standard response is frequently wrong, because it treats the message as an obstacle rather than as information. Understand what is being freed first.
Backward functions need values from the forward pass. Differentiating u² requires knowing u, so PowBackward0 holds a reference to it — and you can see the reference directly:
w = torch.tensor(2.0, requires_grad=True)
u = w * 3
loss = u ** 2
print("saved:", loss.grad_fn._saved_self)
loss.backward()
print("saved:", loss.grad_fn._saved_self)
saved: tensor(6., grad_fn=<MulBackward0>)
RuntimeError: Trying to backward through the graph a second time (or directly access
saved tensors after they have already been freed) ...
There it is. This backward node saved the intermediate value 6.0, and after the backward pass that saved value is no longer available.
Saved activations are an important component of training memory and can dominate parameter memory for some architectures, batch sizes and sequence lengths. That is why releasing values that are no longer needed matters.
Attributes with names such as _saved_self expose useful autograd internals for investigation, and PyTorch’s documentation itself uses them when explaining saved tensors. The exact _saved_* attribute available on a particular backward node is nevertheless implementation-specific, so use this as an inspection technique rather than an application interface. PyTorch frees them as soon as the backward pass is done with them, because for the overwhelmingly common case — one forward, one backward, next batch — holding them any longer is wasted memory.
Note what was not freed. The graph structure is still there:
print(type(loss.grad_fn).__name__)
show_graph(loss.grad_fn)
PowBackward0
PowBackward0
MulBackward0
AccumulateGrad -> leaf tensor, value 2.0
So the shape of the record survives and the values it needs to differentiate do not. That is a useful precision: “the graph was freed” is not quite what happened, and it explains why your inspection tools still work after a backward pass while a second backward() does not.
Which turns the error message into a question about your program:
Why is this code traversing the same graph twice?
There are three common answers, and they call for three different responses.
A new forward pass should have built a new graph. This is a common case, and unnecessary retain_graph=True can retain saved state longer than needed and substantially increase memory use. The better repair is usually to understand why stale forward state is being reused and construct the intended fresh graph:
for batch in batches:
loss = compute_loss(model, batch) # new graph
loss.backward() # traversed once, freed
step(params)
If a loss computed outside the loop is being differentiated inside it, or a cached loss from a previous iteration is being reused, retaining the graph makes the exception disappear while the program continues to differentiate a stale computation.
The algorithm genuinely needs two backward passes through one graph. This does happen: some gradient-penalty terms, some meta-learning updates, and any case where you need two different gradients of the same forward computation. Then retain_graph=True is the correct tool and you should expect to pay for it in memory.
State is being carried across iterations. This is the interesting one, because it is where “just add retain_graph=True” changes the mathematics rather than merely the memory. Consider a loop that carries a hidden value forward, and measure the depth of the graph each step:
def depth(fn):
if fn is None:
return 0
return 1 + max([depth(n) for n, _ in fn.next_functions] or [0])
w = torch.tensor(0.9, requires_grad=True)
hidden = torch.tensor(1.0)
for step in range(5):
hidden = hidden * w + 0.1
loss = hidden ** 2
print(f"step={step} depth={depth(loss.grad_fn)}")
loss.backward(retain_graph=True)
print(f"w.grad: {w.grad.item():.4f}")
step=0 depth=4
step=1 depth=6
step=2 depth=8
step=3 depth=10
step=4 depth=12
w.grad: 26.2882
The graph grows by two nodes per step and never stops. Every step’s saved activations are still alive, because the current graph still references them. Over a long sequence this is exactly the shape of a memory leak, and it is one of the more common ways a training run dies at step 4,000 having been fine at step 40.
Detaching the carried state instead gives a graph of constant depth:
w = torch.tensor(0.9, requires_grad=True)
hidden = torch.tensor(1.0)
for step in range(5):
hidden = hidden * w + 0.1
loss = hidden ** 2
print(f"step={step} depth={depth(loss.grad_fn)}")
loss.backward()
hidden = hidden.detach()
print(f"w.grad: {w.grad.item():.4f}")
step=0 depth=4
step=1 depth=4
step=2 depth=4
step=3 depth=4
step=4 depth=4
w.grad: 10.0000
Both versions ran. Neither raised. And they produced different gradients: 26.29 and 10.00.
That difference is the whole point, so be precise about what each one computed. The first differentiated each step’s loss through the entire history back to the beginning, and accumulated all five results — which is exactly the gradient of the sum of the five losses with respect to w. Confirm it independently:
w = torch.tensor(0.9, requires_grad=True)
h = torch.tensor(1.0)
total = 0
for _ in range(5):
h = h * w + 0.1
total = total + h ** 2
total.backward()
print(f"{w.grad.item():.4f}")
26.2882
The same number. The second version differentiated each loss through one step only, discarding the dependence on earlier steps.
Both computations are meaningful, but the mathematical choice is made by whether the carried state remains connected to its earlier history.
In the first version, hidden remains connected across steps, so each later loss depends on the complete preceding history. retain_graph=True makes repeated backward traversals through that retained history possible; it is not, by itself, what defines the full-history algorithm.
In the second version, hidden.detach() deliberately cuts that history after every step, giving a one-step form of truncated backpropagation through time. Truncation is common in recurrent or state-carrying sequence models when retaining the full history would be too expensive, but it is not a universal description of how all long-sequence models are trained.
The important lesson is that adding retain_graph=True can enable code whose graph structure has a mathematical meaning you have not yet examined. If you add it because a message told you to, you have chosen one of these without knowing that a choice was on offer.
Understand why the graph needs to survive before asking PyTorch to keep it. The flag has a memory cost, and in a loop that carries state it also has a mathematical meaning.
In-place operations and the values backward needs
Two things you will read about in-place operations are both wrong: that they break autograd, and that they are fine because PyTorch would tell you. The truth is narrower and more useful, and it follows directly from the saved tensors we just looked at.
Here is one in-place operation used twice, in computations that differ only in the consumer:
w = torch.tensor([2.0], requires_grad=True)
u = w * 3
v = u + 1
u.add_(10)
v.sum().backward()
print("case A, w.grad =", w.grad.item())
case A, w.grad = 3.0
w = torch.tensor([2.0], requires_grad=True)
u = w * 3
v = u ** 2
u.add_(10)
v.sum().backward()
RuntimeError: one of the variables needed for gradient computation has been modified
by an inplace operation: [torch.FloatTensor [1]], which is output 0 of Add, is at
version 1; expected version 0 instead.
Same in-place call, opposite outcomes. The difference is what the consuming operation needs in order to differentiate itself:
w = torch.tensor([2.0], requires_grad=True)
u = w * 3
print("AddBackward0 saved a copy of its input:", hasattr((u + 1).grad_fn, "_saved_self"))
print("PowBackward0 saved:", (u ** 2).grad_fn._saved_self)
AddBackward0 saved a copy of its input: False
PowBackward0 saved: tensor([6.], grad_fn=<MulBackward0>)
The derivative of u + 1 with respect to u is 1, so AddBackward0 needs nothing from the forward pass and does not care what happens to u afterwards. The derivative of u² is 2u, so PowBackward0 saved u, and modifying u invalidates it.
PyTorch detects this rather than silently computing a wrong gradient. Every tensor carries a version counter, incremented by each in-place modification, and every saved tensor records the version it saw:
w = torch.tensor([2.0], requires_grad=True)
u = w * 3
print("u._version:", u._version)
u.add_(10)
print("u._version:", u._version)
u._version: 0
u._version: 1
At backward time the recorded version is compared against the current one. That is the entire mechanism, and it explains the error message word for word: is at version 1; expected version 0.
It also explains a detail that misdirects people. The message says the modified tensor is output 0 of Add — Add, not Pow. After u.add_(10), the operation that most recently produced u is the in-place addition, so that is how autograd describes it. The message identifies the tensor that was modified, by its current provenance. It does not name the node whose saved value was invalidated. When you read one of these, the tensor is the clue; the operation name in the message is where it ended up, not where the conflict is.
The practical rule that follows is not a prohibition. PyTorch supports many in-place operations, but its own autograd documentation discourages relying on them merely as a memory optimization: autograd already reuses buffers aggressively, and the memory benefit of an in-place spelling is often smaller than expected.
What matters for us is not whether in-place operations are categorically good or bad, but why one particular mutation is accepted while another invalidates backward.
x = torch.tensor([-1.0, 2.0], requires_grad=True)
h = x * 2
h = torch.relu_(h)
h.sum().backward()
print("x.grad:", x.grad)
x.grad: tensor([0., 2.])
That works because ReLU’s backward pass needs its output to decide which entries pass a gradient, and the output is the modified tensor. nn.ReLU(inplace=True) and the various fused in-place operations in real models are not living dangerously.
So the useful formulation is:
One important autograd failure occurs when an in-place operation modifies a tensor whose earlier value is required by a later backward computation. PyTorch tracks these mutations with version counters and raises when the saved value can no longer be trusted.
There are additional restrictions around in-place operations, including operations performed directly on leaf tensors that require gradients and mutations involving aliased views. So “was this tensor saved for backward?” explains the examples above, but it is not the complete rule for every legal in-place operation in PyTorch.
And the diagnostic signature is specific enough to search for: an error that mentions a version number, arriving during backward(), naming a tensor whose shape you can usually match to a particular intermediate. From there, look for in-place operations — +=, -=, *=, methods ending in an underscore, slice assignments, and writes through views or detached tensors that share storage — on the path between the forward operation that saved the value and the backward call.
If the invalidated tensor is hard to identify in a large model, torch.autograd.detect_anomaly() will report the forward operation associated with the failing backward node, which usually locates it immediately:
with torch.autograd.detect_anomaly():
loss = compute_loss()
loss.backward()
It adds substantial diagnostic overhead and is meant for investigation rather than normal training. It earns its place here because it is one of PyTorch’s most convenient built-in ways to associate a backward failure with the forward operation that created the problematic node. Chapter 11 uses it again for non-finite gradients.
The record is built by the operations that actually run
One assumption is worth dislodging explicitly, because it is the natural one to bring from other frameworks and from the way model code looks on the page. In ordinary eager PyTorch execution, autograd does not define one fixed differentiation graph when the model class is written. The backward graph is constructed from the operations that actually execute during that forward pass, which means ordinary Python control flow can change the graph from one invocation to the next:
def f(w, x):
y = w * x
if y.sum() > 0:
return y ** 2
return y.abs()
w = torch.tensor(2.0, requires_grad=True)
for x in (torch.tensor(1.0), torch.tensor(-1.0)):
out = f(w, x)
print(f"x={x.item():+.0f} grad_fn={type(out.grad_fn).__name__} "
f"gradient={torch.autograd.grad(out, w)[0].item():+.1f}")
x=+1 grad_fn=PowBackward0 gradient=+4.0
x=-1 grad_fn=AbsBackward0 gradient=+1.0
Same function, same parameter, two different graphs and two different gradients, because a branch that depended on the data chose different operations. Loops whose length depends on the input, early exits, masking that varies per batch, and layers applied conditionally all have the same property.
Three consequences follow, and each one is something you will use later.
The forward pass and graph construction are the same event, which is why you cannot inspect a model’s autograd structure without running it, and why every diagnostic in this chapter needed an actual forward pass first.
A graph is valid for the iteration that built it. Reusing one across iterations is the situation from the previous section, now with a reason rather than a rule.
And a bug can exist on a path that today’s data did not take. A batch that triggers the other branch can produce a gradient failure in code that has been running for a week. When you cannot reproduce an autograd error, the branch structure of the forward pass is a good place to look, and the failing batch is worth saving.
Chapter 13 returns to this from the opposite direction, when torch.compile tries to capture graphs ahead of time and control flow becomes the reason it gives up.
Using AI on autograd failures
Autograd is a domain where an assistant can produce a plausible, syntactically flawless repair that removes the visible symptom while changing the computation.
Setting requires_grad_(True) on a disconnected tensor can create a new gradient starting point without reconnecting its earlier history. Using retain_graph=True can permit later traversals only if the required graph state was retained before it would otherwise have been released. Replacing an in-place operation may remove a version-counter error while still leaving the surrounding computation conceptually wrong.
Each suggestion can be legitimate. None should be accepted until you know which mechanism actually failed. All three change the symptom. Whether any of them restores the computation you intended is a different question, and it is not one the error message contains enough information to answer.
So ask for a structural claim you can check before asking for a change:
Here is a PyTorch computation. Parameter p ends up with p.grad is None
after loss.backward().
Do not modify the code and do not propose a fix yet.
1. Trace the path from p to loss, operation by operation. For each
intermediate tensor, predict requires_grad, is_leaf, whether it should
have a grad_fn, and whether its .grad will be populated by default
after backward().
2. Identify the first operation after which the result no longer has a
differentiable path back to p, and name the mechanism: a tensor that
never required gradients, a detach, a no_grad region, a value extracted
to Python or NumPy, or a name rebound to a different tensor.
3. State which of your predictions would change if your diagnosis is wrong,
and give me the smallest set of print statements that distinguishes your
explanation from the alternatives.
Then stop.
The value is in the second and third parts. Asking for the boundary forces a commitment to one location rather than a list of things that are sometimes true about missing gradients. Asking which predictions would change if the diagnosis is wrong forces the answer to be falsifiable, so that running the prints either confirms the mechanism or tells you which assumption to investigate next.
When the boundary is confirmed, the follow-up is worth phrasing carefully too:
The break is at <operation>. Before changing it, tell me what the detach
there was doing for the program: what breaks if I remove it, and is there
a place where the value is still needed in detached form?
Because a detach() in real code is often doing a job. Sometimes it is in the wrong place, as it was in the loop that opened this chapter, and sometimes the log or the target or the frozen branch that depends on it will quietly break when you take it out.
The rule from Chapters 1 and 2 has not changed: ask for predictions about the program before asking for a rewrite of the program, and accept a fix only when you can name the quantity it corrected and observe the correction.
What you should now be able to answer
Here is a fragment of the kind you will meet in a real codebase — a two-stage model where someone has already noticed that encoder_scale.grad was None, and has already added a line intended to fix it.
encoder_scale = torch.tensor(2.0, requires_grad=True)
head_weight = torch.tensor(0.5, requires_grad=True)
x = torch.tensor([1.0, 2.0, 3.0])
target = torch.tensor([2.0, 4.0, 6.0])
with torch.no_grad():
features = encoder_scale * x
features.requires_grad_(True) # added to "fix" the missing gradient
prediction = head_weight * features
loss = ((prediction - target) ** 2).mean()
loss.backward()
encoder_scale requires_grad=True is_leaf=True grad_fn=None grad=None
head_weight requires_grad=True is_leaf=True grad_fn=None grad=tensor(-18.6667)
features requires_grad=True is_leaf=True grad_fn=None grad=tensor([-0.3333, -0.6667, -1.0000])
prediction requires_grad=True is_leaf=False grad_fn=MulBackward0 grad=None
loss requires_grad=True is_leaf=False grad_fn=MeanBackward0 grad=None
Which tensors participate in autograd? features, head_weight, prediction, the subtraction, the squaring and the mean. x and target do not require gradients. encoder_scale requires gradients but takes no part in this graph: the multiplication that used it ran inside torch.no_grad() and was never recorded.
Which are leaves? encoder_scale, head_weight, x, target — and features, which is the one worth pausing on. It was produced by an operation, but an unrecorded one, so it has no grad_fn and enters the graph as an input rather than as a computed value. prediction, the error and the loss are non-leaves.
Which have a grad_fn? prediction (MulBackward0), the subtraction (SubBackward0), the squaring, and the loss (MeanBackward0). Everything else has None, including features.
Where will .grad be populated? head_weight.grad and features.grad. Both are leaves that require gradients and both are reachable from the loss. encoder_scale.grad stays None.
Did the added line fix anything? No, and this is the case the chapter has been building toward. features.requires_grad_(True) made features a new leaf tensor that requires gradients, so backward() now populates features.grad with real, finite, correctly-shaped numbers. That does not by itself make features a model parameter or guarantee that any update step will modify it. That does not by itself make features a model parameter or guarantee that any update step will modify it. A gradient appeared. The symptom is gone. And encoder_scale.grad is still None, because declaring that gradients should be tracked from features forward cannot reconstruct a path backwards through an operation that was never recorded. The line created a new starting point in the middle of the model rather than reconnecting the old one.
Where is the first broken edge? At the no_grad block. encoder_scale requires gradients on the line before; features has no grad_fn on the line after. Everything downstream of that point is intact, which is exactly why editing anything downstream — as the added line did — cannot help.
Is that a bug? It depends entirely on intent, which is why this is the right question rather than the answer. If the encoder is intentionally frozen, running its forward computation without recording autograd history can be exactly what you want. But in that case the added features.requires_grad_(True) line is usually unnecessary if the only trainable state is downstream in the head: it creates and retains a gradient with respect to features even though nothing intends to update features itself. If the encoder was meant to be fine-tuned, this silently trains nothing but the head, and the only visible symptom is a model that plateaus.
What would distinguish those two cases from outside? Whether encoder_scale changes over training. Snapshot it, run a few steps, and compare. A parameter that is intended to be frozen and one that is accidentally disconnected produce identical loss curves and identical .grad values; the difference is in what the author intended, and the only way to check is to establish which parameters the program is actually training and compare that against what it claims to train.
If the encoder should train, what is the repair? Remove the no_grad block, and remove the line that was added to compensate for it. Not encoder_scale.requires_grad_(True), which is already true. And the evidence that the repair worked is not merely that some downstream gradient exists — features.grad already gave us one of those. The relevant evidence is that encoder_scale.grad is now populated and that encoder_scale actually changes when the update step runs.
Why did backward() succeed? Because loss.requires_grad is True, which it is as soon as any single tensor on the path requires gradients. backward() running successfully is evidence that something is connected, and no evidence at all about what.
Exercises
These are written to be run, and they map onto the notebook that accompanies this chapter.
-
Derive the graph before you inspect it. For
w = torch.tensor(3.0, requires_grad=True),a = w + 1,b = a * w,c = b ** 2, draw the record by hand, predict everyrequires_grad,is_leafandgrad_fn, and predictdc/dwanalytically. Then runshow_graph(c.grad_fn)andc.backward()and check all of it. Note thatwappears twice; explain from the printed graph why its two contributions add. -
Watch the gradient that is not retained. Using the same computation, use a hook on
bto observe the gradient flowing into it duringbackward(), then repeat withretain_grad(), then repeat withtorch.autograd.grad(). Confirm all three report the same number, and state which of them left.gradfields untouched. -
Two cuts that look the same. Build a computation with two trainable leaves, and produce a version where one path is removed with
detach()and a version where the same path is removed by computing it insidetorch.no_grad(). Find a printed property that distinguishes the two situations. Then construct a case where they are not interchangeable, using a single operation with two operands. -
The trap the loss cannot detect. Take the opening loop and confirm, in the broken version, that
loss.requires_gradisTrueandloss.grad_fnexists at every step. Then write the smallest check that would have failed. Argue for whether it belongs in the training loop permanently or only during investigation. -
Backward twice, on purpose. Construct a computation where two separate losses share an intermediate, and compute gradients for both. Do it with
retain_graph=True, then again with two forward passes, then again with a singletorch.autograd.gradcall taking both outputs. Compare the resulting gradients and explain any differences. -
Make the graph grow. Run the hidden-state loop from this chapter for 200 steps with
retain_graph=True, recording the graph depth. Predict the depth at step 200 before running. Then detach the state and confirm the depth is constant. Explain what would eventually go wrong with the first version on a long sequence. -
Find the saved tensor. As a version-specific inspection exercise, construct an in-place modification that autograd accepts and one that it rejects, differing only in the consuming operation. Inspect an available
_saved_*attribute on the consumer’sgrad_fnand the tensor’s_versionbefore and after. Then explain the operation named in the error message.
Treat the underscore-prefixed attributes in this exercise as diagnostic windows into the current implementation, not APIs that application code should depend on.
-
A branch that only sometimes exists. Write a forward pass whose graph depends on a property of the input batch, and construct one batch that produces a working gradient and one that produces a disconnected parameter. Then write the assertion that would catch it on any batch.
-
Audit an unfamiliar model. Take any model you did not write, run one forward and backward pass, and use
reachable_leavesto list which of its parameters the loss can actually reach. Compare that list against the parameters you believed were being trained.
Next: build the network
We now have the three layers this book started with. Chapter 1 established what training does: forward, backward, update, as three separable phases with different failure modes. Chapter 2 established what flows through it: tensors with shapes that carry meaning nothing enforces. This chapter established what PyTorch recorded while those tensors flowed — a graph of operations, built as they executed, holding references to the leaves it will accumulate into and to the values it will need in order to differentiate.
Those three are enough to build a neural network without any of PyTorch’s neural-network machinery, and that is the next chapter. Weights created directly as trainable leaves; a forward pass written as explicit matrix multiplications and activations; a loss implemented by hand and checked against PyTorch’s own; backward() populating gradients on parameters you created; and an update loop you write yourself.
Everything in it is a mechanism from these three chapters, at slightly larger scale. The parameters are the leaves whose AccumulateGrad nodes we just walked. The graph is rebuilt on every iteration, which is why the loop works at all. And the failure modes are the ones you now have evidence for: a parameter that is not connected, a gradient that accumulated when it should have been cleared, a shape that broadcast when it should have raised.
Then, once the network exists and works, we replace the hand-written parts with nn.Module, nn.Linear and an optimizer — and those will not be new machinery. They will be a way of organizing tensors, leaves and gradients that we have already built by hand.
A neural network, without nn.Module.
