Autograd: What Did PyTorch Record, and Where Does the Gradient Stop?

Sat, 08 Aug 2026 12:55:00 +0100

Here is a training loop with two parameters. It runs, it does not warn, and the loss goes to zero.

import torch

x = torch.tensor([1.0, 2.0, 3.0, 4.0])
target = torch.tensor([3.0, 6.0, 9.0, 12.0])

w1 = torch.tensor(0.5, requires_grad=True)
w2 = torch.tensor(1.0, requires_grad=True)
params = [w1, w2]

feature_log = []

def stage_one(t):
    f = w1 * t
    detached = f.detach()          # keep the values for later inspection
    feature_log.append(detached)
    return detached

learning_rate = 0.02

for step in range(400):
    f = stage_one(x)
    prediction = w2 * f
    loss = ((prediction - target) ** 2).mean()

    loss.backward()

    with torch.no_grad():
        for p in params:
            if p.grad is not None:
                p -= learning_rate * p.grad
                p.grad.zero_()

    if step in (0, 9, 99, 399):
        print(f"step={step:3d} loss={loss.item():.6f} "
              f"w1={w1.item():.4f} w2={w2.item():.4f}")

step=  0 loss=46.875000 w1=0.5000 w2=1.3750
step=  9 loss=11.521060 w1=0.5000 w2=3.7071
step= 99 loss=0.000009 w1=0.5000 w2=5.9979
step=399 loss=0.000000 w1=0.5000 w2=6.0000

Read the columns the way Chapter 1 taught. The loss falls to zero, which is the outcome we asked for. And w1 never moves. Not slowly, not by a little: it holds 0.5000 for four hundred steps while w2 climbs to exactly 6.0, which is the value that makes w1 * w2 equal to the 3 we were trying to learn. Half the model was frozen, the other half absorbed the entire job, and the objective was satisfied anyway.

Detach on Programmer.ie

Autograd: What Did PyTorch Record, and Where Does the Gradient Stop?