00: What Are We Actually Doing?

PyTorch From First Principles

This is Step 0 of PyTorch From First Principles.

The book starts with the smallest possible training problem and ends with a small GPT-style language model built from scratch.

But the goal is not to memorize a collection of PyTorch commands.

The goal is to understand the machinery underneath those commands well enough that unfamiliar PyTorch code becomes something you can inspect, reason about, debug and change.

This is not a survey of every PyTorch feature.

It is a practical path through the parts that make the framework make sense:

tensors and shapes
gradients and autograd
manual training loops
nn.Module and recursive model structure
Dataset, DataLoader and transforms
CNN geometry
high-dimensional feature spaces
attention and transformer shapes
models that refuse to learn
GPU performance and memory
torch.compile and compiler behavior
reproducible experiments and regressions
one complete GPT-style model

Some later chapters are advanced interludes. You do not need to master every compiler guard or high-dimensional geometric detail on the first pass.

The core promise is simpler:

By the end of the book, you should be able to look at a PyTorch system and ask the right questions about what it is actually doing.

Before we can do that, there is one question underneath everything else:

What are we actually doing when we train a model?

The shortest possible explanation

A neural network is a function with adjustable numbers.

We give it some input.

It produces an output.

We measure how wrong that output is.

Then we adjust the numbers so that next time it is slightly less wrong.

That is the core loop.

    flowchart TD
    A[Input] --> B[Model fθ]
    B --> C[Prediction]
    C --> D[Compare with answer]
    D --> E[Loss]
    E --> F[Compute gradients ∇Loss]
    F --> G[Adjust parameters]
    G --> B
  

Deep learning becomes complicated because modern models may contain billions of adjustable numbers and enormous chains of mathematical operations.

The underlying idea does not change.

PyTorch helps us:

represent the numbers
perform the calculations
record the relationships needed for differentiation
calculate gradients
organize model state
move work across devices
feed data into the model
measure and optimize execution

The rest of this book will unpack those responsibilities one layer at a time.

Start with a function

Forget neural networks for a moment.

Suppose the world follows this simple rule:

y = 3x

If x is 2, the answer is 6.

But imagine we do not know that the multiplier is 3.

We only know that our model looks like this:

y = wx

The value w is a parameter we want to learn.

Start with a bad guess:

w = 1

For x = 2, our model predicts:

2 × 1 = 2

The correct answer is 6, so the model is wrong.

Now we need some way of measuring how wrong it is.

One simple loss function is squared error:

loss = (prediction - answer)²

For our prediction:

loss = (2 - 6)²
     = 16

Training means finding a change to w that reduces that loss.

We could try random values until something works, but there is a much better method.

We can calculate the gradient.

The gradient tells us how the loss changes when we change a parameter.

That is the first major idea behind training neural networks.

Now do it in PyTorch

Install PyTorch using the installation command appropriate for your operating system and hardware from the official PyTorch site, then open Python and try this:

import torch

x = torch.tensor([2.0])
w = torch.tensor([1.0], requires_grad=True)   # we want to learn w
target = torch.tensor([6.0])

prediction = x * w
loss = (prediction - target) ** 2

loss.backward()   # compute gradients

print("prediction:", prediction.item())
print("loss:", loss.item())
print("gradient:", w.grad.item())

You should get:

prediction: 2.0
loss: 16.0
gradient: -16.0

There is a lot happening in those few lines.

The most important line is:

loss.backward()

PyTorch works backwards through the operations that produced loss and calculates how the loss changes with respect to w.

That value ends up here:

w.grad

For this example the gradient is -16.

At the current value of w, a sufficiently small increase in w will reduce the loss.

So we can change w slightly in that direction.

Take one learning step

Let’s update the parameter manually.

learning_rate = 0.1

with torch.no_grad():   # we don't want to track this update as part of the graph
    w -= learning_rate * w.grad

print(w)

The new value of w is:

tensor([2.6000], requires_grad=True)

Our original guess was 1. After one step PyTorch has moved it toward the correct value, 3.

Run the calculation again with w = 2.6:

prediction = 2 × 2.6
           = 5.2

The prediction has moved from 2 to 5.2.

The correct answer is 6.

One gradient step made the model much better.

That is training.

Not metaphorically.

That is the mechanism we will keep scaling up.

A complete training loop

Now repeat the process in a loop.

import torch

x = torch.tensor([2.0])
target = torch.tensor([6.0])
w = torch.tensor([1.0], requires_grad=True)

learning_rate = 0.1

for step in range(10):
    prediction = x * w
    loss = (prediction - target) ** 2

    print(
        f"step={step:02d} "
        f"w={w.item():.6f} "
        f"prediction={prediction.item():.6f} "
        f"loss={loss.item():.6f}"
    )

    loss.backward()

    with torch.no_grad():
        w -= learning_rate * w.grad

    w.grad.zero_()

Now each printed row describes one coherent state: the value of w, the prediction produced by that value, and the corresponding loss.

You should see w rapidly approach 3 and the loss approach zero.

Notice the sequence:

predict
   ↓
measure error
   ↓
calculate gradients
   ↓
update parameters
   ↓
repeat

Those operations contain the skeleton of a real neural-network training loop.

Later we will replace the single parameter w with thousands, millions, or potentially billions of parameters.

We will replace x * w with layers of matrix multiplication, nonlinear activations, convolutions, attention and transformer blocks.

We will replace the manual parameter update with an optimizer such as SGD or AdamW.

Conceptually, though, we will still be doing the same thing.

Visualising progress: tracking the loss

A list of numbers is useful.

A learning curve often tells you more.

Here is the same experiment with the loss recorded after each forward pass:

import torch
import matplotlib.pyplot as plt

x = torch.tensor([2.0])
target = torch.tensor([6.0])
w = torch.tensor([1.0], requires_grad=True)
lr = 0.1
num_steps = 20
losses = []

for step in range(num_steps):
    prediction = x * w
    loss = (prediction - target) ** 2

    losses.append(loss.item())

    loss.backward()

    with torch.no_grad():
        w -= lr * w.grad

    w.grad.zero_()

plt.plot(range(num_steps), losses, marker="o")
plt.xlabel("Step")
plt.ylabel("Loss")
plt.title("Learning w = 3 with one data point")
plt.grid(True)
plt.show()

You should see the loss fall toward zero as w approaches 3.

This habit becomes important later.

When a model stops learning, a curve can reveal plateaus, instability, divergence and regressions that a single final number hides.

The first object underneath everything: the tensor

You may have noticed that we did not use ordinary Python numbers.

We wrote:

x = torch.tensor([2.0])

A tensor is the fundamental numerical object in PyTorch.

Inputs, parameters, activations, predictions and losses can all be represented as tensors.

For now, one fact is enough:

x = torch.tensor([2.0])
print(x.shape)

produces:

torch.Size([1])

Soon we will work with shapes such as:

[B, F]
[B, C, H, W]
[B, T, D]
[B, H, T, T]

Those shapes are not decoration.

They describe what the data means and which operations are valid.

That is why Step 01 is devoted to tensor shapes before we build larger models.

What PyTorch remembers

Consider this:

x = torch.tensor([2.0])
w = torch.tensor([3.0], requires_grad=True)

z = x * w
y = z ** 2

We can picture the forward computation as:

    graph LR
    w((w)) --> mul((×))
    x((x)) --> mul
    mul --> z[z = x·w]
    z --> sq((²))
    sq --> y[y = z²]
  

When gradient tracking is enabled, PyTorch records the information needed to differentiate the result with respect to the tensors that require gradients.

Then:

y.backward()

propagates gradients backward through those recorded dependencies.

For now, remember this:

PyTorch does not merely store numbers. It can also track the mathematical relationships between operations on those numbers.

Step 02 will open that mechanism properly.

Neural networks are the same idea at a different scale

A neuron can be written roughly as:

output = activation(inputs × weights + bias)

A layer performs many such operations together.

A network composes layers.

Training calculates how the loss depends on the parameters and updates them accordingly.

So when you eventually see:

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 10)

I do not want that code to mean:

“PyTorch neural-network incantation.”

I want you to see learnable tensors organized into modules and used in operations whose gradients PyTorch can calculate.

That distinction becomes central in Step 04, where we will see that a large PyTorch model is mostly simple objects composed inside other simple objects, with PyTorch recursively walking the resulting structure.

The model is only part of the training system

The current book goes further than the model itself because real failures often occur outside forward().

A useful picture is:

raw data
   ↓
transforms
   ↓
Dataset / DataLoader
   ↓
model
   ↓
loss
   ↓
autograd
   ↓
optimizer
   ↓
updated model

And once the model is large enough, another layer appears:

hardware
memory
profiling
compiler graphs
reproducibility
measurement

That is why later chapters spend so much time on debugging and observation.

A program that runs without an exception can still be wrong, slow, unstable or worse than yesterday’s version.

The book’s recurring rule is:

Make the hidden structure visible before guessing.

Sometimes that means printing a tensor shape.

Sometimes it means checking a gradient.

Sometimes it means inspecting registered parameters, transformed samples, GPU memory, compiler recompilations or two experiment runs side by side.

Three things to understand before moving on

You do not need to understand all of calculus before using PyTorch.

But three ideas from this chapter should be clear.

1. A model contains parameters

These are numbers whose values can change during training.

In our tiny model, w was the parameter.

2. A loss measures error

The loss converts “how good was the prediction?” into a quantity we can optimize.

Our loss was:

(prediction - target)²

3. Gradients tell us how to change the parameters

PyTorch’s automatic differentiation calculates how the loss depends on the parameters.

An optimization rule then uses those gradients to update the parameters.

Everything else in the book elaborates on those three ideas and on the structures needed to make them work at scale.

Challenge: make PyTorch discover another number

Before moving to Step 01, change the problem.

Suppose:

y = 5x

Create training examples such as:

x = 1 → y = 5
x = 2 → y = 10
x = 3 → y = 15
x = 4 → y = 20

Start w at a deliberately bad value and train it.

Can you get PyTorch to discover that:

w ≈ 5

Then try changing the learning rate:

0.001
0.01
0.1
1.0

Do not just look for the value that works.

Watch what happens to the loss.

Ask:

Does it fall smoothly?
Does it move too slowly?
Does it oscillate?
Does it explode?

That is already the beginning of debugging a training system.

The current path through PyTorch From First Principles

The book now follows this sequence:

00  What Are We Actually Doing?
01  Tensor Shapes and Shape Debugging
02  Autograd and Gradient Debugging
03  Build a Neural Network Without nn.Module
04  Recursive Composition — How PyTorch Organizes Models
05  DataLoader Performance and Input-Pipeline Debugging
05A Transforms — From Raw Data to the Tensor the Model Sees
06  CNN Shape Debugging
06A Beyond 3D — High-Dimensional Tensors and an SVM
07  Attention Shapes, Q/K/V, Heads and Masks
08  Model Not Learning? Systematic Debugging
09  CUDA and Performance Debugging
09A Compiler Debugging — Graph Breaks, Guards and Recompiles
09B Training Regressions and Reproducible Experiments
10  Build a Small GPT-Style Language Model From Scratch

The inserted chapters are there because the book grew as we identified concepts that deserved to stand on their own.

05A makes the preprocessing boundary explicit.

06A teaches how to reason in feature spaces that humans cannot visualize.

09A opens the compiler rather than treating torch.compile() as a switch.

09B deals with a different class of failure: nothing crashes, but the new run is worse.

The final GPT chapter is the capstone because it forces all of these ideas to meet in one working system.

What proficiency will look like by the end

You do not need to remember every PyTorch function name.

You should instead be able to ask questions like:

What tensor is this?
What does each dimension mean?
What dtype and device is it on?
What transform produced it?
Which parameters belong to this model?
Does the gradient reach them?
Did the optimizer actually update them?
Can the model overfit a tiny batch?
Where is the memory going?
What is starving the accelerator?
Did torch.compile capture the graph I expected?
Why did it recompile?
Is this run actually worse than the baseline?
Can I reproduce the comparison?

That is the kind of PyTorch understanding this book is trying to build.

We start with one parameter because every larger system needs somewhere simple to begin.

Next we examine the object underneath nearly everything PyTorch does:

the tensor.