Count the parameters a mixture layer actually uses
Route a four-token batch through four experts, count the selected experts, and compare active parameters with the layer's total.
A mixture-of-experts layer contains many expert networks but evaluates only the top-k selected by a router for each token. The model’s total parameter count includes every expert; the active count for a token includes the router plus the selected experts. Those are different numbers, and confusing them turns a sparse model into an apparently dense one.
This example builds a layer with four experts of dimension 8 and routes a batch
of four tokens with top-k 2. The router has 8 * 4 = 32 weights. Each expert has
8 * 8 = 64 weights. Active parameters per token are 32 + 2 * 64 = 160; total
parameters are 32 + 4 * 64 = 288. The run also prints which experts each token
selected, so the arithmetic can be checked against the routing decisions.
Run it
From the canonical repository at commit
f958967d70992d7b09d9c75000bd8c543f1644d4:
python examples/ch19/moe.py --dim 8 --experts 4 --top-k 2 --jsonPython 3.11 or later with PyTorch. CPU-only, seeded, no model download, network, GPU or credentials. It finishes in about a second.
Implementation
The router produces one logit per expert. Softmax turns them into weights, top-k selects the experts, and the selected weights are renormalised before their outputs are summed. The code iterates over the batch, so each token’s selection is explicit.
class MoELayer(nn.Module):
def __init__(self, dim: int, num_experts: int, top_k: int = 2) -> None:
super().__init__()
if top_k > num_experts:
raise ValueError("top_k must not exceed num_experts")
self.dim = dim
self.num_experts = num_experts
self.top_k = top_k
self.router = nn.Linear(dim, num_experts, bias=False)
self.experts = nn.ModuleList([Expert(dim) for _ in range(num_experts)])
def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, List[List[int]]]:
# x: [batch, dim]
logits = self.router(x) # [batch, num_experts]
probs = torch.softmax(logits, dim=-1)
topk_vals, topk_idx = torch.topk(probs, self.top_k, dim=-1)
# Normalize selected.
topk_vals = topk_vals / topk_vals.sum(dim=-1, keepdim=True)
out = torch.zeros_like(x)
selected: List[List[int]] = []
for b in range(x.shape[0]):
idxs = topk_idx[b].tolist()
selected.append(idxs)
for rank, expert_idx in enumerate(idxs):
out[b] = out[b] + topk_vals[b, rank] * self.experts[expert_idx](x[b:b + 1]).squeeze(0)
return out, logits, selectedExpected outcome
Observed with the command above (exit 0). Selected stable fields from the end of the JSON:
"active_parameters": 160,
"total_parameters": 288,
"top_k": 2,
"num_experts": 4The full output also contains the four output vectors, the four router-logit
vectors and the two expert ids selected for each token. The active fraction is
160 / 288 = 0.556, so this tiny layer is not very sparse. The chapter’s larger
Mixtral-shaped example is where the active fraction becomes small; the principle
is the same, but the number is not transferable.
Mechanism and limitations
Documented: the layer rejects top_k > num_experts, computes router logits,
softmaxes them, selects top-k, renormalises the selected weights and sums the
selected expert outputs. run counts active parameters as the router plus
top_k copies of one expert, and total parameters as the router plus every
expert. The tests check the selected count, the active/total relationship, the
output shape and JSON serialisation.
Observed: the command exited 0 with active_parameters: 160,
total_parameters: 288, top_k: 2 and num_experts: 4. The targeted
tests.test_ch19_moe is part of the canonical suite; no independent test count is
claimed here.
Not shown: a trained router, real expert parallelism, communication cost, capacity limits, load balancing, or a production MoE model. The layer processes one token at a time in a Python loop; its latency is not a kernel benchmark. The active-parameter count is an accounting identity for this architecture, not a measurement of what the hardware read.
Challenge this result
Run with --top-k 1 and --top-k 4. Compute active parameters by hand for each
before running. At top-k 4 every expert is active, so the layer should behave
like a dense mixture; explain why the active count equals the total only in that
case. Then change --experts 8 and compute the active fraction at top-k 2.
Finally, explain why a lower active fraction does not automatically mean lower
latency: routing, gather/scatter and load imbalance also cost time.
Source and verification
Canonical source:
examples/ch19/moe.py
at commit f958967d70992d7b09d9c75000bd8c543f1644d4. Tests:
tests/test_ch19_moe.py
.
Chapter: Mixture-of-Experts Inference
.
The run above is a fresh local execution on Windows 11 with Python 3.13.15 and
PyTorch 2.6.0+cu124; it uses a randomly initialised layer, not a trained model.
Understand this example
Copy this prompt into your AI tool. No code runs here.
Apply this example
Copy this prompt into your AI tool. No code runs here.