Models From First Principles 06: Inside Tiny — Residual Blocks, Attention and Sparse Autoencoders

Sat, 08 Aug 2026 15:00:00 +0100

Inside Tiny: Residual Blocks, Attention and Sparse Autoencoders

In the previous post, we built a compact recursive model around one idea:

context + candidate + latent state
              ↓
          projection
              ↓
         reusable core
              ↓
       proposed update
              ↓
      z ← z + α · update
              ↓
            repeat

That architecture looked more sophisticated than MR.Q, EBT or SICQL because it introduced recurrence.

But the central idea of this series is that a model stops looking mysterious when we keep opening it.

PyTorch Attention Shapes: Q, K, V, Multi-Head Attention Masks and Transformer Dimension Errors

Sat, 08 Aug 2026 13:30:00 +0100

PyTorch: Zero to Hero — Step 07

Attention code is where tensor-shape mistakes stop being annoying and start becoming architectural.

A CNN usually makes its dimensional assumptions fairly obvious. Attention does not.

A tensor that starts as:

(batch, sequence, embedding)

is projected into Q, K and V, split into heads, transposed, multiplied, masked, normalized, multiplied again, transposed again, concatenated and projected back to the embedding dimension.

A single bad view, transpose, mask shape or head calculation can produce anything from an immediate runtime error to a model that trains while attending to the wrong tokens.

Attention on Programmer.ie: Modern AI programming

Models From First Principles 06: Inside Tiny — Residual Blocks, Attention and Sparse Autoencoders

Inside Tiny: Residual Blocks, Attention and Sparse Autoencoders

PyTorch Attention Shapes: Q, K, V, Multi-Head Attention Masks and Transformer Dimension Errors

PyTorch: Zero to Hero — Step 07