Models From First Principles 03: SICQL — Building a Model From Q, V and Policy Networks
SICQL — Building a Model From Q, V and Policy Networks
In the previous post we took the MR.Q idea and expanded it into something richer.
Instead of asking one question of a shared representation, EBT asked several:
How good is this state-action pair? -> Q
How good is the state more generally? -> V
What action should be preferred? -> Policy
How much better is Q than V? -> Advantage
That already gave us a more expressive system.
But there is another architectural step we can take.
We can stop treating those outputs as anonymous branches inside one large model and make them first-class models in their own right.
That is the central idea of this post.
We are going to build a SICQL-style architecture where the system is visibly composed from smaller models:
context embedding + candidate embedding
↓
encoder
↓
zsa
┌─────────┼─────────┐
↓ ↓ ↓
QHead VHead PolicyHead
↓ ↓ ↓
Q V action logits
└──── Q - V ────────┘
advantage
The important lesson is not the acronym.
The important lesson is composition.
A model can be built from models.
And once those components are explicit, we can inspect them, test them, train them, replace them and reason about them independently.
1. Why make the heads explicit?
At first glance, these two designs may look almost identical.
Inline design
class Model(nn.Module):
def __init__(self, d):
super().__init__()
self.encoder = nn.Linear(d * 2, 256)
self.q = nn.Linear(256, 1)
self.v = nn.Linear(256, 1)
self.pi = nn.Linear(256, 3)
Component design
class Model(nn.Module):
def __init__(self, encoder, q_head, v_head, policy_head):
super().__init__()
self.encoder = encoder
self.q_head = q_head
self.v_head = v_head
self.policy_head = policy_head
Mathematically, either might represent the same functions.
Architecturally, they are not the same.
The second design gives names and boundaries to the components.
That matters because a boundary gives us something we can test.
input contract
↓
component
↓
output contract
If QHead promises:
[B, Z] -> [B]
we can test that promise without involving the rest of the system.
If PolicyHead promises:
[B, Z] -> [B, A]
we can replace it without rewriting the encoder.
The architecture becomes easier to reason about because the code now reflects the conceptual model.
2. Start with the latent representation
We will assume that we already have two embeddings:
context_emb # [B, D]
candidate_emb # [B, D]
A simple pair encoder can concatenate them and project them into a latent state-action representation.
import torch
from torch import nn
class PairEncoder(nn.Module):
def __init__(self, embedding_dim: int, latent_dim: int):
super().__init__()
self.net = nn.Sequential(
nn.Linear(embedding_dim * 2, latent_dim),
nn.ReLU(),
nn.LayerNorm(latent_dim),
nn.Linear(latent_dim, latent_dim),
nn.ReLU(),
)
def forward(self, context, candidate):
assert context.shape == candidate.shape
assert context.dim() == 2
pair = torch.cat([context, candidate], dim=-1)
return self.net(pair)
Shape flow:
context [B, D]
candidate [B, D]
↓ concatenate
pair [B, 2D]
↓ encoder
zsa [B, Z]
We will call that final representation zsa.
The name is useful shorthand for encoded state-action information, but remember the rule from the previous posts:
A variable name does not prove semantics.
If both context and candidate are present, then zsa is certainly a pair representation.
Whether it deserves a stronger reinforcement-learning interpretation depends on how the data and objective are defined.
3. Build QHead as its own model
The Q head predicts one scalar from the latent representation.
class QHead(nn.Module):
def __init__(self, latent_dim: int, hidden_dim: int):
super().__init__()
self.net = nn.Sequential(
nn.Linear(latent_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1),
)
def forward(self, zsa):
return self.net(zsa).squeeze(-1)
That is the entire model.
zsa [B, Z]
↓
Linear(Z -> H)
↓
ReLU
↓
Linear(H -> 1)
↓
Q [B]
Nothing mystical happened.
We simply gave the scalar predictor a name and a contract.
Now we can test it independently.
def test_q_head_shape():
head = QHead(latent_dim=64, hidden_dim=32)
z = torch.randn(8, 64)
q = head(z)
assert q.shape == (8,)
That small test has real value.
It tells us that if the composed model later emits the wrong Q shape, the error is probably outside the Q head.
4. Build VHead independently
The V head can have exactly the same physical architecture while representing a different learned function.
class VHead(nn.Module):
def __init__(self, latent_dim: int, hidden_dim: int):
super().__init__()
self.net = nn.Sequential(
nn.Linear(latent_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1),
)
def forward(self, z):
return self.net(z).squeeze(-1)
Notice something important.
The code does not know whether this is Q or V.
Physically both heads are:
vector
↓
MLP
↓
scalar
Their meaning comes from:
which inputs they receive
+
which target they are trained against
+
which loss is applied
This is one of the most important recurring ideas in model architecture.
Two networks may be structurally identical while learning entirely different functions.
5. The state-only V question returns
In the EBT post we raised a subtle issue.
Suppose the encoder receives:
context + candidate
and then V receives the resulting pair representation.
Then the path is:
(context, candidate)
↓
zsa
↓
VHead
↓
value
That value is candidate-conditioned.
Calling it a pure V(s) may therefore be too strong.
A stricter architecture uses two representations:
context ───────────────> state encoder ──> zs ──> VHead
context + candidate ───> pair encoder ───> zsa ─> QHead
└─────> PolicyHead
We can implement that explicitly.
class StateEncoder(nn.Module):
def __init__(self, embedding_dim, latent_dim):
super().__init__()
self.net = nn.Sequential(
nn.Linear(embedding_dim, latent_dim),
nn.ReLU(),
nn.LayerNorm(latent_dim),
)
def forward(self, context):
return self.net(context)
Then:
zs = state_encoder(context)
zsa = pair_encoder(context, candidate)
v = v_head(zs)
q = q_head(zsa)
Now the architectural contract matches the conventional notation more closely:
V(s)
Q(s, a)
Does that automatically make the model better?
No.
It makes the semantics cleaner.
Whether it improves the actual task is an empirical question.
That distinction matters.
6. Build PolicyHead as a classifier
The policy head predicts logits over actions.
class PolicyHead(nn.Module):
def __init__(self, latent_dim: int, hidden_dim: int, num_actions: int):
super().__init__()
self.net = nn.Sequential(
nn.Linear(latent_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, num_actions),
)
def forward(self, zsa):
return self.net(zsa)
Shape flow:
zsa [B, Z]
↓
PolicyHead
↓
action logits [B, A]
If there are three possible actions:
reject
revise
accept
then:
logits.shape == [B, 3]
We deliberately return logits, not probabilities.
For training with cross entropy:
loss_policy = F.cross_entropy(action_logits, action_targets)
PyTorch expects raw logits.
If we need probabilities for inspection:
probs = action_logits.softmax(dim=-1)
Again, separate the training interface from the presentation interface.
7. Compose the model
Now we have four independent pieces:
PairEncoder
QHead
VHead
PolicyHead
We can compose them.
class InContextQModel(nn.Module):
def __init__(self, encoder, q_head, v_head, policy_head):
super().__init__()
self.encoder = encoder
self.q_head = q_head
self.v_head = v_head
self.policy_head = policy_head
def forward(self, context, candidate):
zsa = self.encoder(context, candidate)
q = self.q_head(zsa)
v = self.v_head(zsa)
policy_logits = self.policy_head(zsa)
advantage = q - v
return {
"zsa": zsa,
"q": q,
"v": v,
"advantage": advantage,
"policy_logits": policy_logits,
}
The whole architecture is now visually obvious.
PairEncoder
│
zsa
┌───────────┼───────────┐
│ │ │
▼ ▼ ▼
QHead VHead PolicyHead
│ │ │
q v logits
│ │
└──────┬────┘
▼
advantage
This is what building a model from models looks like in ordinary PyTorch.
8. Why component boundaries are valuable
Once the heads are separate modules, several engineering practices become much easier.
Independent initialization
q_head = QHead(128, 64)
v_head = VHead(128, 64)
policy_head = PolicyHead(128, 64, 3)
Independent tests
def test_policy_head():
head = PolicyHead(128, 64, 3)
z = torch.randn(4, 128)
logits = head(z)
assert logits.shape == (4, 3)
Independent checkpointing
torch.save(model.q_head.state_dict(), "q_head.pt")
torch.save(model.v_head.state_dict(), "v_head.pt")
Independent replacement
model.q_head = BiggerQHead(...)
Independent freezing
for p in model.v_head.parameters():
p.requires_grad = False
Independent optimization
optimizer = torch.optim.AdamW([
{"params": model.encoder.parameters(), "lr": 1e-4},
{"params": model.q_head.parameters(), "lr": 3e-4},
{"params": model.v_head.parameters(), "lr": 3e-4},
{"params": model.policy_head.parameters(), "lr": 1e-4},
])
Those are not merely code-organization conveniences.
They create experimental control.
9. First-class heads enable clean ablations
Suppose we want to know whether the policy objective actually helps Q ranking.
With explicit heads we can train:
Experiment A:
encoder + Q
Experiment B:
encoder + Q + V
Experiment C:
encoder + Q + V + Policy
Then compare the metric we actually care about.
results = {
"q_only": evaluate(model_q),
"q_v": evaluate(model_qv),
"q_v_policy": evaluate(model_full),
}
This is much stronger than assuming:
more heads = more intelligence.
Perhaps the policy head regularizes the representation and improves ranking.
Perhaps it causes gradient conflict and makes Q worse.
Perhaps it does nothing.
The architecture gives us a hypothesis.
The experiment gives us the answer.
10. Explicit components reveal parameter ownership
A surprisingly common debugging problem is not knowing which parameters belong to which conceptual subsystem.
With first-class heads:
for name, p in model.named_parameters():
print(name, tuple(p.shape))
might produce:
encoder.net.0.weight
encoder.net.0.bias
encoder.net.2.weight
...
q_head.net.0.weight
q_head.net.0.bias
...
v_head.net.0.weight
...
policy_head.net.0.weight
...
The namespace itself becomes documentation.
We can count parameters by subsystem.
def count_params(module):
return sum(p.numel() for p in module.parameters())
print("encoder:", count_params(model.encoder))
print("Q:", count_params(model.q_head))
print("V:", count_params(model.v_head))
print("policy:", count_params(model.policy_head))
This lets us answer a useful question:
Where did the model complexity actually go?
Often the shared encoder dominates.
The extra heads may be cheap.
11. Initialize the heads deliberately
Because the heads are independent modules, we can make initialization explicit too.
def init_linear(module):
if isinstance(module, nn.Linear):
nn.init.xavier_normal_(module.weight)
nn.init.zeros_(module.bias)
q_head.apply(init_linear)
v_head.apply(init_linear)
policy_head.apply(init_linear)
Now initialization is not an accidental side effect of construction.
It is part of the model contract.
That matters especially when loading partial checkpoints or comparing experiments.
12. A full standalone SICQL-style model
Here is the whole system in one place.
import torch
from torch import nn
import torch.nn.functional as F
class PairEncoder(nn.Module):
def __init__(self, embedding_dim, latent_dim):
super().__init__()
self.net = nn.Sequential(
nn.Linear(embedding_dim * 2, latent_dim),
nn.ReLU(),
nn.LayerNorm(latent_dim),
nn.Linear(latent_dim, latent_dim),
nn.ReLU(),
)
def forward(self, context, candidate):
assert context.dim() == 2
assert context.shape == candidate.shape
return self.net(torch.cat([context, candidate], dim=-1))
class ScalarHead(nn.Module):
def __init__(self, latent_dim, hidden_dim):
super().__init__()
self.net = nn.Sequential(
nn.Linear(latent_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1),
)
def forward(self, z):
return self.net(z).squeeze(-1)
class PolicyHead(nn.Module):
def __init__(self, latent_dim, hidden_dim, num_actions):
super().__init__()
self.net = nn.Sequential(
nn.Linear(latent_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, num_actions),
)
def forward(self, z):
return self.net(z)
class SICQLStyleModel(nn.Module):
def __init__(self, embedding_dim=128, latent_dim=128, hidden_dim=64, num_actions=3):
super().__init__()
self.encoder = PairEncoder(embedding_dim, latent_dim)
self.q_head = ScalarHead(latent_dim, hidden_dim)
self.v_head = ScalarHead(latent_dim, hidden_dim)
self.policy_head = PolicyHead(latent_dim, hidden_dim, num_actions)
def forward(self, context, candidate):
zsa = self.encoder(context, candidate)
q = self.q_head(zsa)
v = self.v_head(zsa)
policy_logits = self.policy_head(zsa)
return {
"q": q,
"v": v,
"advantage": q - v,
"policy_logits": policy_logits,
"zsa": zsa,
}
Run it:
model = SICQLStyleModel()
context = torch.randn(16, 128)
candidate = torch.randn(16, 128)
out = model(context, candidate)
for key, value in out.items():
print(key, value.shape)
Expected:
q [16]
v [16]
advantage [16]
policy_logits [16, 3]
zsa [16, 128]
At this point we have an inspectable composed model.
Now we need to decide how to train it.
13. Q objective: pairwise preference
Assume we have:
context
preferred candidate
rejected candidate
We want:
Q(context, preferred) > Q(context, rejected)
A stable pairwise logistic objective is:
def pairwise_q_loss(q_good, q_bad):
return F.softplus(-(q_good - q_bad)).mean()
Training:
good = model(context, preferred)
bad = model(context, rejected)
loss_q = pairwise_q_loss(good["q"], bad["q"])
The quantity being optimized is the difference.
That means absolute Q values are not automatically calibrated.
Again:
ranking and calibration are different problems.
14. V objective: expectile regression
A common way to train a value estimate is expectile regression.
Define the residual:
residual = target_q - v
Then give asymmetric weight to positive and negative residuals.
def expectile_loss(v, target_q, tau=0.7):
diff = target_q.detach() - v
weight = torch.where(diff > 0, tau, 1 - tau)
return (weight * diff.square()).mean()
The parameter tau controls which part of the Q distribution the V model is pushed toward.
For example:
tau = 0.5 -> ordinary squared-error center
tau > 0.5 -> places more weight on underestimation
The exact semantics depend on the training setup.
The important point for this series is architectural:
V has its own objective.
The fact that Q and V share an encoder does not make them the same model.
15. Policy objective
Suppose action labels are integers:
0 = reject
1 = revise
2 = accept
Then:
loss_policy = F.cross_entropy(
out["policy_logits"],
action_targets,
)
If we have behavioral or preference weights, the policy objective could be more sophisticated.
But the basic architecture is simply a classifier over the shared representation.
16. Compose the losses
A simple multi-task objective could be:
loss = (
loss_q
+ 0.5 * loss_v
+ 0.2 * loss_policy
)
These coefficients are not magic constants.
They are hyperparameters expressing how strongly each task influences the shared representation.
That immediately creates another experimental question:
Are the tasks helping each other or fighting each other?
17. Measure gradient conflict between heads
Because the encoder is shared, each objective sends gradients into the same parameters.
We can inspect the direction of those gradients.
def flat_grad(loss, params):
grads = torch.autograd.grad(
loss,
params,
retain_graph=True,
allow_unused=True,
)
parts = []
for p, g in zip(params, grads):
if g is None:
parts.append(torch.zeros_like(p).flatten())
else:
parts.append(g.flatten())
return torch.cat(parts)
Then:
params = list(model.encoder.parameters())
g_q = flat_grad(loss_q, params)
g_v = flat_grad(loss_v, params)
g_pi = flat_grad(loss_policy, params)
Cosine similarity:
def cosine(a, b):
return F.cosine_similarity(a, b, dim=0).item()
print("Q vs V:", cosine(g_q, g_v))
print("Q vs policy:", cosine(g_q, g_pi))
print("V vs policy:", cosine(g_v, g_pi))
Interpretation:
positive -> gradients broadly cooperate
near zero -> mostly unrelated
negative -> objectives push encoder in opposing directions
This does not automatically tell us what to do.
It tells us what is happening.
That is the evidence-first approach.
18. Advantage is a derived quantity
We compute:
advantage = q - v
That operation has no parameters.
Q model ──┐
├── subtraction -> advantage
V model ──┘
This distinction matters.
Advantage is not another neural network.
It is a derived signal produced by two learned models.
If advantage behaves badly, the cause is upstream.
That gives us a useful debugging path:
bad advantage
↓
inspect Q
↓
inspect V
↓
inspect shared representation
↓
inspect objectives/data
19. Detach advantage or not?
Sometimes advantage is used only for telemetry or downstream decision logic.
Then we may want:
advantage = (q - v).detach()
That makes a strong statement:
Nothing downstream of
advantageis allowed to send gradients back through Q or V.
Compare:
advantage = q - v
versus:
advantage = (q - v).detach()
These have identical numeric values in the forward pass.
But their optimization behavior is different.
This is exactly the sort of small line that an LLM can generate correctly-looking code around while changing the training semantics dramatically.
When in doubt:
print(advantage.requires_grad)
print(advantage.grad_fn)
Runtime evidence wins.
20. Policy weights are not policy probabilities
An easy interpretability mistake is to look directly at the final linear layer weights and treat them as the policy.
Suppose:
last = model.policy_head.net[-1]
print(last.weight.shape)
The weight matrix might be:
[A, H]
Those are model parameters.
They are not the action probabilities for a specific example.
The policy for an example is:
logits = model(...)["policy_logits"]
probs = logits.softmax(dim=-1)
Parameters tell us how the model transforms representations.
Outputs tell us what it predicts for the current input.
Do not confuse the two.
21. Checkpoint the components separately
Explicit composition makes partial checkpoints natural.
torch.save({
"encoder": model.encoder.state_dict(),
"q_head": model.q_head.state_dict(),
"v_head": model.v_head.state_dict(),
"policy_head": model.policy_head.state_dict(),
}, "sicql_components.pt")
Loading:
state = torch.load("sicql_components.pt", map_location="cpu")
model.encoder.load_state_dict(state["encoder"])
model.q_head.load_state_dict(state["q_head"])
model.v_head.load_state_dict(state["v_head"])
model.policy_head.load_state_dict(state["policy_head"])
This lets us ask more interesting questions.
For example:
What happens if we keep the encoder fixed but retrain Q?
What happens if we transfer Q and replace the policy head?
What happens if V is trained on a different dataset?
Those experiments are easy because the architecture has boundaries.
22. But component checkpoints create compatibility contracts
Modularity creates flexibility.
It also creates compatibility problems.
Suppose an old Q head expects:
latent_dim = 128
but the new encoder outputs:
latent_dim = 256
Then:
q = old_q_head(new_encoder_output)
fails.
Or worse, suppose the dimensions match but the meaning of the representation changed because the encoder was retrained.
The shapes still work.
The semantics may not.
A robust checkpoint should therefore store metadata.
checkpoint = {
"model_version": 3,
"embedding_model": "example-embed-v2",
"embedding_dim": 768,
"latent_dim": 128,
"encoder_version": "pair-v4",
"head_versions": {
"q": "q-v2",
"v": "v-v1",
"policy": "pi-v3",
},
"state_dict": model.state_dict(),
}
Compatibility is more than tensor shape.
23. Test components before testing the full model
A useful test ladder is:
1. individual layer
2. individual head
3. encoder
4. composed forward pass
5. individual loss
6. combined loss
7. backward pass
8. optimizer update
9. tiny-batch overfit
10. validation behavior
For example:
def test_model_contract():
model = SICQLStyleModel(
embedding_dim=32,
latent_dim=16,
hidden_dim=8,
num_actions=3,
)
context = torch.randn(5, 32)
candidate = torch.randn(5, 32)
out = model(context, candidate)
assert out["q"].shape == (5,)
assert out["v"].shape == (5,)
assert out["advantage"].shape == (5,)
assert out["policy_logits"].shape == (5, 3)
assert out["zsa"].shape == (5, 16)
This turns architecture documentation into executable code.
24. Verify every subsystem receives gradients
After one backward pass:
loss.backward()
we can inspect each component.
def grad_summary(module):
total = 0.0
missing = []
for name, p in module.named_parameters():
if p.grad is None:
missing.append(name)
else:
total += p.grad.norm().item()
return total, missing
Then:
for name, module in {
"encoder": model.encoder,
"q": model.q_head,
"v": model.v_head,
"policy": model.policy_head,
}.items():
norm, missing = grad_summary(module)
print(name, norm, missing)
This gives us an immediate answer to:
Did every component actually participate in training?
A head can exist in the architecture and still be functionally disconnected from the loss.
25. Verify parameters actually move
Gradient presence is not enough.
Save a copy before step().
before = {
name: p.detach().clone()
for name, p in model.named_parameters()
}
optimizer.step()
Then:
for name, p in model.named_parameters():
delta = (p.detach() - before[name]).abs().max().item()
print(name, delta)
Now we know which components changed.
This catches mistakes such as:
- parameters omitted from the optimizer;
- frozen modules;
- zero gradients;
- stale optimizer references after replacing a head.
That last one is especially important.
26. Replacing a head can invalidate the optimizer
Suppose we build the optimizer:
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
Then later replace Q:
model.q_head = BiggerQHead(...)
The new Q head’s parameters are not automatically added to the existing optimizer.
The optimizer still holds references to the old parameter objects.
Check ownership:
opt_ids = {
id(p)
for group in optimizer.param_groups
for p in group["params"]
}
for name, p in model.q_head.named_parameters():
print(name, id(p) in opt_ids)
If false, rebuild the optimizer or update its parameter groups.
This is a perfect example of why architectural modularity and runtime debugging belong together.
27. Swap Q without changing the rest of the model
Now we can exploit the interface.
class DeeperQHead(nn.Module):
def __init__(self, latent_dim, hidden_dim):
super().__init__()
self.net = nn.Sequential(
nn.Linear(latent_dim, hidden_dim),
nn.GELU(),
nn.Linear(hidden_dim, hidden_dim),
nn.GELU(),
nn.Linear(hidden_dim, 1),
)
def forward(self, z):
return self.net(z).squeeze(-1)
Replacement:
model.q_head = DeeperQHead(128, 64)
Nothing else needs to change because the interface remains:
[B, 128] -> [B]
That is the power of the component boundary.
28. Replace the encoder instead
The same principle works higher in the hierarchy.
Our model only requires the encoder to satisfy:
(context, candidate) -> zsa
So we could replace concatenation with richer interaction features.
class InteractionEncoder(nn.Module):
def __init__(self, d, z):
super().__init__()
self.net = nn.Sequential(
nn.Linear(d * 4, z),
nn.ReLU(),
nn.LayerNorm(z),
)
def forward(self, context, candidate):
features = torch.cat([
context,
candidate,
context - candidate,
context * candidate,
], dim=-1)
return self.net(features)
Then:
model.encoder = InteractionEncoder(128, 128)
The heads do not care.
This is architecture as substitution.
29. Stable interfaces let us run controlled experiments
We can compare:
Encoder A: concat
Encoder B: concat + difference + product
Encoder C: attention-based
while keeping:
QHead
VHead
PolicyHead
training objective
optimizer
training data
fixed.
That is a clean encoder ablation.
Similarly we can compare head architectures while keeping the encoder fixed.
Modularity makes scientific control easier.
30. Candidate reranking
A common use case is one context with many candidate responses.
Suppose:
context.shape == [B, D]
candidates.shape == [B, K, D]
We can repeat the context:
B, K, D = candidates.shape
ctx = context[:, None, :].expand(B, K, D)
ctx = ctx.reshape(B * K, D)
cand = candidates.reshape(B * K, D)
Run all candidates:
out = model(ctx, cand)
q = out["q"].reshape(B, K)
Choose the highest Q:
best_idx = q.argmax(dim=-1)
But now SICQL gives us more than Q.
We can also inspect:
v = out["v"].reshape(B, K)
adv = out["advantage"].reshape(B, K)
policy = out["policy_logits"].reshape(B, K, -1)
The system has become a diagnostic surface, not merely a ranker.
31. Multiple outputs can disagree
Suppose candidate A has:
high Q
small advantage
policy says revise
and candidate B has:
slightly lower Q
large advantage
policy says accept
Which candidate wins?
The neural architecture cannot answer that question for us.
That is a decision policy layered on top of the model outputs.
For example:
def decision_score(out):
accept_prob = out["policy_logits"].softmax(-1)[..., 2]
return (
0.7 * out["q"]
+ 0.2 * out["advantage"]
+ 0.1 * accept_prob
)
Those weights are another hypothesis.
They require validation.
The model architecture produces signals.
The application decides how to use them.
Keep those layers conceptually separate.
32. Architecture vs objective vs decision logic
At this point we can distinguish three levels clearly.
Architecture
encoder
QHead
VHead
PolicyHead
Training objectives
pairwise Q loss
expectile V loss
policy cross entropy
Runtime decision logic
rank by Q
rank by advantage
accept if policy probability > threshold
combine signals
These are different things.
A lot of model confusion comes from mixing them together.
33. A small synthetic training experiment
Let’s build a toy task so the complete system can be executed.
We will generate contexts and candidates where the preferred candidate is the one more aligned with the context.
def make_batch(batch_size=256, dim=32):
context = torch.randn(batch_size, dim)
good = context + 0.25 * torch.randn(batch_size, dim)
bad = -context + 0.25 * torch.randn(batch_size, dim)
action_good = torch.full((batch_size,), 2, dtype=torch.long)
action_bad = torch.zeros(batch_size, dtype=torch.long)
return context, good, bad, action_good, action_bad
Build the model:
model = SICQLStyleModel(
embedding_dim=32,
latent_dim=64,
hidden_dim=32,
num_actions=3,
)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
Training loop:
for step in range(1000):
context, good, bad, a_good, a_bad = make_batch()
out_good = model(context, good)
out_bad = model(context, bad)
loss_q = pairwise_q_loss(out_good["q"], out_bad["q"])
target_q = torch.cat([
out_good["q"].detach(),
out_bad["q"].detach(),
])
pred_v = torch.cat([
out_good["v"],
out_bad["v"],
])
loss_v = expectile_loss(pred_v, target_q, tau=0.7)
policy_logits = torch.cat([
out_good["policy_logits"],
out_bad["policy_logits"],
])
action_targets = torch.cat([a_good, a_bad])
loss_policy = F.cross_entropy(policy_logits, action_targets)
loss = loss_q + 0.5 * loss_v + 0.2 * loss_policy
optimizer.zero_grad(set_to_none=True)
loss.backward()
optimizer.step()
if step % 100 == 0:
with torch.no_grad():
rank_acc = (out_good["q"] > out_bad["q"]).float().mean()
print(
step,
float(loss),
float(rank_acc),
)
This is not intended as a benchmark.
It is an executable architecture demonstration.
34. Run the baseline first
Before celebrating the composed model, measure a trivial baseline.
Because our synthetic good candidates are deliberately close to the context, cosine similarity may solve the task almost perfectly.
def cosine_score(context, candidate):
return F.cosine_similarity(context, candidate, dim=-1)
Evaluate:
base_good = cosine_score(context, good)
base_bad = cosine_score(context, bad)
baseline_acc = (base_good > base_bad).float().mean()
If the simple baseline wins, that is not embarrassing.
It tells us the neural model is unnecessary for that dataset.
Complexity should earn its place.
35. A component should justify itself
For every component ask:
What new capability is this intended to provide?
What metric should improve if it works?
What baseline does it need to beat?
What ablation isolates its contribution?
For VHead:
Does it improve downstream policy learning?
Does its advantage signal correlate with useful decisions?
Does removing it change performance?
For PolicyHead:
Does policy supervision improve the shared representation?
Does it hurt Q ranking?
Does the policy itself predict useful actions?
For the pair encoder:
Does learned interaction outperform cosine similarity or a linear probe?
That is how architecture becomes evidence rather than decoration.
36. The deeper lesson: software architecture matters inside neural architecture
SICQL introduces something beyond another mathematical idea.
It introduces software architecture into model architecture.
The conceptual system says:
Q
V
Policy
The code says:
QHead
VHead
PolicyHead
That alignment matters.
When conceptual boundaries and code boundaries match, we gain:
- clearer tests;
- clearer checkpoints;
- clearer metrics;
- cleaner ablations;
- easier replacement;
- easier debugging;
- easier reuse.
A model is not only a function.
It is also a piece of software.
37. When not to split everything into modules
There is a danger in taking modularity too far.
You could turn every linear layer into its own class.
LinearOne
ActivationOne
LinearTwo
OutputWrapper
That would not necessarily improve comprehension.
A useful component boundary should correspond to something meaningful:
separate objective
separate semantic role
separate interface
separate experiment
separate checkpoint
separate replacement candidate
QHead qualifies.
A single ReLU usually does not.
Good modularity compresses thought.
Bad modularity creates ceremony.
38. Why SICQL still isn’t enough
We have now built a model composed from several models.
But all of them still operate in one forward pass.
The pattern is:
input
↓
encoder
↓
heads
↓
outputs
There is no persistent internal state that is refined over several reasoning steps.
The network does not repeatedly revisit a latent representation.
That creates the next architectural question:
What if a single feed-forward transformation is not enough?
We can introduce recurrence.
But we can go further than one recurrent state.
We can create two coupled states operating at different scales:
low-level state
↕
high-level state
The low-level state can update several times while the high-level state changes more slowly.
That is the architectural move we will make next.
What we learned
SICQL gave us another layer of decomposition.
We started with:
one model
Then discovered:
encoder
Q model
V model
policy model
And each of those decomposed again into:
Linear
activation
Linear
The important advance was not simply adding more layers.
It was making the conceptual components explicit.
That gave us:
independent tests
independent losses
independent checkpoints
independent replacement
clean ablations
parameter ownership
clear debugging paths
The model became more complex mathematically while becoming easier to inspect structurally.
That is a useful architecture.
In the next post we make a much larger jump.
Next: HRM — Hierarchical Reasoning With Fast and Slow Recurrent State
Instead of evaluating the representation once, we will build two coupled recurrent systems:
input
↓
low-level recurrent processing
↓
repeated local refinement
↓
high-level recurrent update
↓
repeat
That is where Models From First Principles moves from composed feed-forward models into iterative reasoning architectures.
