Example

One model, four sub-models, and one shared tensor

Build the chapter's multi-head evaluator and its recurrent block, print the shape at every stage of the diagram, and count the parameters of each part.

The word model suggests one monolithic object, and the code usually tells a different story: a complicated model is a collection of smaller models, each a collection of layers. Opening an unfamiliar model means descending that hierarchy until the tensors are visible.

This program builds the chapter’s multi-head evaluator and the recurrent block from it: the shape at every stage of the diagram, the parameter count of every part, and what six applications of a block cost.

Run it

Save this as model_inside_model.py and run it with Python 3.11 or later. It needs PyTorch: pip install torch, then python model_inside_model.py.

"""The model inside the model: one representation, several small models."""
import torch
from torch import nn

EMB, HID, ACTIONS = 16, 8, 3


def head(dim, out_dim):
    return nn.Sequential(nn.Linear(dim, dim // 2), nn.ReLU(), nn.Linear(dim // 2, out_dim))


class MultiHeadEvaluator(nn.Module):
    def __init__(self, embedding_dim=EMB, hidden_dim=HID, num_actions=ACTIONS):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(embedding_dim * 2, hidden_dim), nn.ReLU(), nn.LayerNorm(hidden_dim),
            nn.Linear(hidden_dim, hidden_dim),
        )
        self.q_head = head(hidden_dim, 1)
        self.v_head = head(hidden_dim, 1)
        self.policy_head = head(hidden_dim, num_actions)

    def forward(self, context, response):
        pair = torch.cat([context, response], dim=-1)
        z = self.encoder(pair)
        q = self.q_head(z).squeeze(-1)
        v = self.v_head(z).squeeze(-1)
        return {"pair": pair, "z": z, "q": q, "v": v, "advantage": q - v, "policy_logits": self.policy_head(z)}


class RecurrentBlock(nn.Module):
    def __init__(self, input_dim, hidden_dim):
        super().__init__()
        self.cell = nn.GRUCell(input_dim, hidden_dim)
        self.norm = nn.RMSNorm(hidden_dim)

    def forward(self, previous_state, current_input):
        return self.norm(self.cell(current_input, previous_state))


torch.manual_seed(0)
model = MultiHeadEvaluator()
context, response = torch.randn(4, EMB), torch.randn(4, EMB)
out = model(context, response)

print("the diagram, proven by shapes")
print(f"  context {tuple(context.shape)} + response {tuple(response.shape)} -> pair {tuple(out['pair'].shape)}")
print(f"  pair -> z {tuple(out['z'].shape)}")
print(f"  z -> q {tuple(out['q'].shape)}  v {tuple(out['v'].shape)}  advantage {tuple(out['advantage'].shape)}")
print(f"  z -> policy_logits {tuple(out['policy_logits'].shape)}")

print("the model contains models")
for name, module in model.named_children():
    print(f"  {name:<14} {type(module).__name__:<12} {sum(p.numel() for p in module.parameters()):>7,d} parameters")
print(f"  {'total':<14} {'':<12} {sum(p.numel() for p in model.parameters()):>7,d} parameters")

block = RecurrentBlock(HID, HID)
state = torch.zeros(4, HID)
for _ in range(6):
    state = block(state, out["z"])
print(f"recurrent block: {sum(p.numel() for p in block.parameters()):,d} parameters, applied 6 times")
print(f"  state {tuple(state.shape)} after 6 updates, parameter count unchanged")

Expected outcome

the diagram, proven by shapes
  context (4, 16) + response (4, 16) -> pair (4, 32)
  pair -> z (4, 8)
  z -> q (4,)  v (4,)  advantage (4,)
  z -> policy_logits (4, 3)
the model contains models
  encoder        Sequential       352 parameters
  q_head         Sequential        41 parameters
  v_head         Sequential        41 parameters
  policy_head    Sequential        51 parameters
  total                           485 parameters
recurrent block: 440 parameters, applied 6 times
  state (4, 8) after 6 updates, parameter count unchanged

Draw the model, then prove it

The diagram is a hypothesis and the runtime shapes are the evidence. The chapter’s own helper for that:

def parameter_report(model):
    for name, module in model.named_children():
        count = sum(p.numel() for p in module.parameters())
        print(f"{name:24s} {count:12,d}")

The printed shapes confirm that one representation feeds every head: z is (4, 8), and the Q head, the V head and the policy head all read that tensor. The advantage is not a layer: it is the difference of two heads. The counts show where the capacity sits: the encoder holds most of it, and the recurrent block shows that iteration adds compute, not parameters.

Mechanism and limitations

Documented: the chapter draws the model as a tree of sub-models, proves the diagram with shapes (context [B, 1024] and response [B, 1024] become pair [B, 2048], then z [B, 256], then q [B]), and reports parameters per part.

Observed: the run above. pair was (4, 32) and z was (4, 8), and the Q head, V head and policy head produced (4,), (4,) and (4, 3) from that tensor. The encoder held 352 parameters, each value head 41, the policy head 51, and the model 485. The recurrent block held 440 parameters and ran six times.

Not shown. The program composes the chapter’s fragments into one run and uses a smaller width than the chapter’s example, so the counts differ from the ones printed there. Nothing is trained, so the advantage and the policy logits mean nothing yet.

Canonical source , the chapter this example composes. Read the chapter .

Understand this example

Copy this prompt into your AI tool. No code runs here.

Apply this example

Copy this prompt into your AI tool. No code runs here.