PyTorch From First Principles on Programmer.ie

PyTorch Transforms: From Raw Data to the Tensor Your Model Actually Sees

Sat, 22 Aug 2026 11:58:00 +0100

PyTorch From First Principles — Interlude 05A

In the previous chapter we treated the DataLoader as a production system.

It reads samples, prepares batches, moves work through worker processes and tries to keep the accelerator fed.

But there is a question hiding inside that pipeline:

What exactly is a sample by the time the model receives it?

A file on disk is not usually the tensor the model trains on.

10: Build a Small GPT-Style Language Model From Scratch

Sat, 08 Aug 2026 14:03:00 +0100

Build a Small GPT-Style Language Model From Scratch in PyTorch

This is the final post in the PyTorch: Zero to Hero series.

We have spent the previous posts learning the machinery underneath PyTorch:

tensors and shapes;
autograd;
manual neural networks;
nn.Module and parameter registration;
DataLoader performance;
convolutional networks;
attention and masks;
training failures;
CUDA performance and torch.compile.

Now we put it together.

The goal is not to download a pretrained model.

Training Regressions and Reproducible Experiments: When Nothing Crashes but the Model Gets Worse

Sat, 08 Aug 2026 14:00:00 +0100

PyTorch: Zero to Hero — Advanced Step 09B

Most bugs in this book have announced themselves loudly.

A tensor had the wrong shape.

A parameter had no gradient.

A model refused to learn.

CUDA ran out of memory.

A compiled graph broke or recompiled.

But some of the most expensive machine-learning bugs are quieter.

The program still runs.

The tests still pass.

Training still finishes.

And yet:

validation loss is worse
training takes 12% longer
peak VRAM is 20% higher
gradients became less stable
compile count increased
throughput dropped
convergence needs twice as many steps

Nothing failed inside the run.

PyTorch Compiler Debugging: Graph Breaks, Guards, Recompiles and torch.compile

Sat, 08 Aug 2026 13:58:00 +0100

PyTorch: Zero to Hero — Advanced Step 09A

In the previous chapter we treated torch.compile as one tool inside a larger performance-debugging workflow.

That is enough until compilation itself becomes the problem.

Then the questions change.

The model runs in eager mode.

The model may even run when compiled.

But perhaps:

compile time is enormous
first call takes seconds
new shapes keep compiling again
graph breaks appear inside forward()
compiled execution is slower than eager
one backend works and another fails
small code changes produce very different compiler behavior

At that point, torch.compile(model) is not the end of the story.

PyTorch Performance Debugging: CUDA OOM, Slow Training, GPU Utilization and torch.compile

Sat, 08 Aug 2026 13:56:00 +0100

PyTorch: Zero to Hero — Step 09

At this point in the series, the model runs.

That does not mean it runs well.

A training loop can be correct and still waste most of the machine.

A model can fit in memory and still spend half its time waiting on synchronization.

A torch.compile call can make code faster, slower, or simply move the bottleneck somewhere else.

A CUDA out-of-memory error can be caused by the model, the optimizer, activations, fragmentation, a leaked reference, a larger batch, a longer sequence, or an innocent-looking tensor that was kept alive by Python.

PyTorch Model Not Learning? A Systematic Debugging Guide

Sat, 08 Aug 2026 13:51:00 +0100

PyTorch: Zero to Hero — Step 08

Your model runs.

The loss is finite.

Nothing crashes.

And it still does not learn.

This is one of the most frustrating states in machine learning because there is no stack trace telling you what is wrong.

The program is valid Python.

The tensors have legal shapes.

The GPU is busy.

The optimizer is stepping.

And the model is useless.

This post is a systematic way to debug that situation.

PyTorch Attention Shapes: Q, K, V, Multi-Head Attention Masks and Transformer Dimension Errors

Sat, 08 Aug 2026 13:30:00 +0100

PyTorch: Zero to Hero — Step 07

Attention code is where tensor-shape mistakes stop being annoying and start becoming architectural.

A CNN usually makes its dimensional assumptions fairly obvious. Attention does not.

A tensor that starts as:

(batch, sequence, embedding)

is projected into Q, K and V, split into heads, transposed, multiplied, masked, normalized, multiplied again, transposed again, concatenated and projected back to the embedding dimension.

A single bad view, transpose, mask shape or head calculation can produce anything from an immediate runtime error to a model that trains while attending to the wrong tokens.

Beyond 3D: Tensors, 100 Dimensions and an SVM From Scratch

Sat, 08 Aug 2026 13:28:00 +0100

PyTorch: Zero to Hero — Advanced Interlude 06A

There is a point in machine learning where the numbers become difficult for a human being to picture.

One dimension is easy.

Two dimensions are easy.

Three dimensions are still possible to draw.

Then somebody hands you a tensor containing 100 features.

Or 768.

Or 4,096.

And suddenly the language changes.

People start talking about embedding spaces, feature directions, hyperplanes, high-dimensional geometry and latent representations.

PyTorch CNN Shape Errors: Conv2d Output Sizes, Channels, Flatten Bugs and How to Debug Them

Sat, 08 Aug 2026 13:26:00 +0100

PyTorch: Zero to Hero — Step 06

CNN code is usually easy to write.

CNN shape bugs are usually easy to create.

A typical failure looks like this:

RuntimeError: Given groups=1, weight of size [32, 3, 3, 3],
expected input[64, 224, 224, 3] to have 3 channels,
but got 224 channels instead

Or this:

RuntimeError: mat1 and mat2 shapes cannot be multiplied

Or worse: the model runs, but the dimensions are wrong in a way that silently damages the architecture.

PyTorch DataLoader Performance: num_workers, pin_memory, Prefetching and Why Your GPU Is Waiting

Sat, 08 Aug 2026 13:21:00 +0100

PyTorch: Zero to Hero — Step 05

A fast model with a slow input pipeline is still a slow training system.

One of the most common PyTorch performance failures looks like this:

GPU utilization: 20% → 95% → 10% → 90% → 15%

The model is not necessarily slow.

The GPU may simply be waiting for the next batch.

This article is about finding out where the wait is happening.

The Most Important Idea in PyTorch: Recursive Composition

Sat, 08 Aug 2026 13:09:00 +0100

PyTorch: Zero to Hero — Step 04

In the previous post we built a neural network using raw tensors and autograd.

We had weights.

We had biases.

We had a forward calculation.

We had a loss.

We called backward().

We updated the parameters.

Nothing essential was missing from the mathematics of training.

Now we are going to add nn.Module.

But I do not want to introduce it as another PyTorch class whose methods you need to memorize.

Build a Neural Network From Scratch in PyTorch Without nn.Module

Sat, 08 Aug 2026 12:59:00 +0100

PyTorch: Zero to Hero — Step 03

Most PyTorch tutorials begin with something like this:

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(2, 16),
    nn.ReLU(),
    nn.Linear(16, 2),
)

That is useful code.

It is also hiding almost everything interesting.

In this post we are going to build the same kind of neural network using ordinary PyTorch tensors.

No nn.Module.

No nn.Linear.

No torch.optim.Adam.

No optimizer.step().

We will manually create the parameters, write the forward pass, calculate the loss, call autograd, update the weights, zero the gradients, batch the data, evaluate the model and then compare the result with the idiomatic PyTorch version.

PyTorch Autograd Debugging: requires_grad, detach, backward() and NaN Gradients

Sat, 08 Aug 2026 12:55:00 +0100

PyTorch: Zero to Hero — Step 02

In the previous post we treated tensor shapes as a debugging problem rather than a mathematical vocabulary exercise.

We are going to do the same thing with autograd.

If you use PyTorch for any serious amount of time, you eventually see errors like:

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

or:

RuntimeError: Trying to backward through the graph a second time...

or worse:

PyTorch Tensor Shapes: Broadcasting, Reshape, View, Permute and the Errors That Waste Your Time

Sat, 08 Aug 2026 12:45:00 +0100

PyTorch: Zero to Hero — Step 01

Most PyTorch bugs are not really “AI bugs”.

They are shape bugs.

You expected:

[batch, features]

but actually had:

[batch, sequence, features]

You expected two tensors to line up.

They broadcast instead.

You called view() after permute() and got a contiguity error.

You removed a dimension with squeeze() and accidentally removed the batch dimension too.

Or you reached the familiar message:

RuntimeError: The size of tensor a (...) must match the size of tensor b (...)

This article is about becoming dangerous enough with tensors that these errors stop being mysterious.

00: What Are We Actually Doing?

Sat, 08 Aug 2026 12:40:00 +0100

PyTorch From First Principles

This is Step 0 of PyTorch From First Principles.

The book starts with the smallest possible training problem and ends with a small GPT-style language model built from scratch.

But the goal is not to memorize a collection of PyTorch commands.

The goal is to understand the machinery underneath those commands well enough that unfamiliar PyTorch code becomes something you can inspect, reason about, debug and change.

Mon, 01 Jan 0001 00:00:00 +0000

PyTorch: Zero to Hero — Working Ledger

Book intention

Learn PyTorch by understanding what the framework is doing from first principles, then build upward from the core training loop to modern model architectures.

This book is already substantially written. The ledger does not assume that published chapters are editorially complete: the new pass should test the progression, remove avoidable duplication, and preserve the practical debugging character that makes the sequence useful.