Model Architecture on Programmer.ie: Modern AI programming

Models From First Principles 06: Inside Tiny — Residual Blocks, Attention and Sparse Autoencoders

Sat, 08 Aug 2026 15:00:00 +0100

Inside Tiny: Residual Blocks, Attention and Sparse Autoencoders

In the previous post, we built a compact recursive model around one idea:

context + candidate + latent state
              ↓
          projection
              ↓
         reusable core
              ↓
       proposed update
              ↓
      z ← z + α · update
              ↓
            repeat

That architecture looked more sophisticated than MR.Q, EBT or SICQL because it introduced recurrence.

But the central idea of this series is that a model stops looking mysterious when we keep opening it.

Models From First Principles 03: SICQL — Building a Model From Q, V and Policy Networks

Sat, 08 Aug 2026 14:44:00 +0100

SICQL — Building a Model From Q, V and Policy Networks

In the previous post we took the MR.Q idea and expanded it into something richer.

Instead of asking one question of a shared representation, EBT asked several:

How good is this state-action pair?      -> Q
How good is the state more generally?    -> V
What action should be preferred?         -> Policy
How much better is Q than V?              -> Advantage

That already gave us a more expressive system.

Models From First Principles 01: MR.Q — Building a Neural Quality Model From Two Embeddings

Sat, 08 Aug 2026 14:33:00 +0100

MR.Q — Building a Neural Quality Model From Two Embeddings

In the previous post, we established the core idea behind this series:

A complicated model becomes understandable when you recursively decompose it into smaller models, blocks, layers and tensor operations.

Now we build the first real model.

Not a transformer.

Not a giant language model.

Not an agent.

A scorer.

We will take two embeddings:

context embedding
response embedding

combine them, encode the relationship between them, and predict one scalar:

Models From First Principles 00: The Model Inside the Model

Sat, 08 Aug 2026 14:27:00 +0100

The Model Inside the Model

This is the first post in Models From First Principles.

The previous PyTorch: Zero to Hero series worked from the bottom up.

We started with tensors.

Then gradients.

Then nn.Module.

Then data pipelines, convolution, attention, debugging, performance and finally a small GPT-style language model built from scratch.

That series answered:

What are the pieces?

This series asks a different question:

What happens when we start composing those pieces into increasingly sophisticated models?