One model, four sub-models, and one shared tensor
Build the chapter's multi-head evaluator and its recurrent block, print the shape at every stage of the diagram, and count the parameters of each part.
The word model suggests one monolithic object, and the code usually tells a different story: a complicated model is a collection of smaller models, each a collection of layers. Opening an unfamiliar model means descending that hierarchy until the tensors are visible.
This program builds the chapter’s multi-head evaluator and the recurrent block from it: the shape at every stage of the diagram, the parameter count of every part, and what six applications of a block cost.
Run it
Save this as model_inside_model.py and run it with Python 3.11 or later. It
needs PyTorch: pip install torch, then python model_inside_model.py.
"""The model inside the model: one representation, several small models."""
import torch
from torch import nn
EMB, HID, ACTIONS = 16, 8, 3
def head(dim, out_dim):
return nn.Sequential(nn.Linear(dim, dim // 2), nn.ReLU(), nn.Linear(dim // 2, out_dim))
class MultiHeadEvaluator(nn.Module):
def __init__(self, embedding_dim=EMB, hidden_dim=HID, num_actions=ACTIONS):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(embedding_dim * 2, hidden_dim), nn.ReLU(), nn.LayerNorm(hidden_dim),
nn.Linear(hidden_dim, hidden_dim),
)
self.q_head = head(hidden_dim, 1)
self.v_head = head(hidden_dim, 1)
self.policy_head = head(hidden_dim, num_actions)
def forward(self, context, response):
pair = torch.cat([context, response], dim=-1)
z = self.encoder(pair)
q = self.q_head(z).squeeze(-1)
v = self.v_head(z).squeeze(-1)
return {"pair": pair, "z": z, "q": q, "v": v, "advantage": q - v, "policy_logits": self.policy_head(z)}
class RecurrentBlock(nn.Module):
def __init__(self, input_dim, hidden_dim):
super().__init__()
self.cell = nn.GRUCell(input_dim, hidden_dim)
self.norm = nn.RMSNorm(hidden_dim)
def forward(self, previous_state, current_input):
return self.norm(self.cell(current_input, previous_state))
torch.manual_seed(0)
model = MultiHeadEvaluator()
context, response = torch.randn(4, EMB), torch.randn(4, EMB)
out = model(context, response)
print("the diagram, proven by shapes")
print(f" context {tuple(context.shape)} + response {tuple(response.shape)} -> pair {tuple(out['pair'].shape)}")
print(f" pair -> z {tuple(out['z'].shape)}")
print(f" z -> q {tuple(out['q'].shape)} v {tuple(out['v'].shape)} advantage {tuple(out['advantage'].shape)}")
print(f" z -> policy_logits {tuple(out['policy_logits'].shape)}")
print("the model contains models")
for name, module in model.named_children():
print(f" {name:<14} {type(module).__name__:<12} {sum(p.numel() for p in module.parameters()):>7,d} parameters")
print(f" {'total':<14} {'':<12} {sum(p.numel() for p in model.parameters()):>7,d} parameters")
block = RecurrentBlock(HID, HID)
state = torch.zeros(4, HID)
for _ in range(6):
state = block(state, out["z"])
print(f"recurrent block: {sum(p.numel() for p in block.parameters()):,d} parameters, applied 6 times")
print(f" state {tuple(state.shape)} after 6 updates, parameter count unchanged")Expected outcome
the diagram, proven by shapes
context (4, 16) + response (4, 16) -> pair (4, 32)
pair -> z (4, 8)
z -> q (4,) v (4,) advantage (4,)
z -> policy_logits (4, 3)
the model contains models
encoder Sequential 352 parameters
q_head Sequential 41 parameters
v_head Sequential 41 parameters
policy_head Sequential 51 parameters
total 485 parameters
recurrent block: 440 parameters, applied 6 times
state (4, 8) after 6 updates, parameter count unchangedDraw the model, then prove it
The diagram is a hypothesis and the runtime shapes are the evidence. The chapter’s own helper for that:
def parameter_report(model):
for name, module in model.named_children():
count = sum(p.numel() for p in module.parameters())
print(f"{name:24s} {count:12,d}")The printed shapes confirm that one representation feeds every head: z is
(4, 8), and the Q head, the V head and the policy head all read that
tensor. The advantage is not a layer: it is the difference of two heads. The
counts show where the capacity sits: the encoder holds most of it, and the
recurrent block shows that iteration adds compute, not parameters.
Mechanism and limitations
Documented: the chapter draws the model as a tree of sub-models, proves
the diagram with shapes (context [B, 1024] and response [B, 1024] become
pair [B, 2048], then z [B, 256], then q [B]), and reports parameters per
part.
Observed: the run above. pair was (4, 32) and z was (4, 8), and
the Q head, V head and policy head produced (4,), (4,) and (4, 3) from
that tensor. The encoder held 352 parameters, each value head 41, the policy
head 51, and the model 485. The recurrent block held 440 parameters and ran
six times.
Not shown. The program composes the chapter’s fragments into one run and uses a smaller width than the chapter’s example, so the counts differ from the ones printed there. Nothing is trained, so the advantage and the policy logits mean nothing yet.
Canonical source , the chapter this example composes. Read the chapter .
Understand this example
Copy this prompt into your AI tool. No code runs here.
Apply this example
Copy this prompt into your AI tool. No code runs here.