PyTorch Zero to Hero 10: Build a Small GPT-Style Language Model From Scratch

Sat, 08 Aug 2026 14:03:00 +0100

Build a Small GPT-Style Language Model From Scratch in PyTorch

This is the final post in the PyTorch: Zero to Hero series.

We have spent the previous posts learning the machinery underneath PyTorch:

tensors and shapes;
autograd;
manual neural networks;
nn.Module and parameter registration;
DataLoader performance;
convolutional networks;
attention and masks;
training failures;
CUDA performance and torch.compile.

Now we put it together.

The goal is not to download a pretrained model.

PyTorch Attention Shapes: Q, K, V, Multi-Head Attention Masks and Transformer Dimension Errors

Sat, 08 Aug 2026 13:30:00 +0100

PyTorch: Zero to Hero — Step 07

Attention code is where tensor-shape mistakes stop being annoying and start becoming architectural.

A CNN usually makes its dimensional assumptions fairly obvious. Attention does not.

A tensor that starts as:

(batch, sequence, embedding)

is projected into Q, K and V, split into heads, transposed, multiplied, masked, normalized, multiplied again, transposed again, concatenated and projected back to the embedding dimension.

A single bad view, transpose, mask shape or head calculation can produce anything from an immediate runtime error to a model that trains while attending to the wrong tokens.

Transformers on Programmer.ie: Modern AI programming

PyTorch Zero to Hero 10: Build a Small GPT-Style Language Model From Scratch

Build a Small GPT-Style Language Model From Scratch in PyTorch

PyTorch Attention Shapes: Q, K, V, Multi-Head Attention Masks and Transformer Dimension Errors

PyTorch: Zero to Hero — Step 07