Compare a skewed router with a balanced one
Send 100 tokens to four experts under two policies and measure the load, queue and drop consequences of a capacity limit.
A router that sends most tokens to the same expert wastes the other experts and overloads one. A capacity limit turns that imbalance into a concrete outcome: tokens wait or are dropped. Load variance is one summary, but the maximum load and the drop count are the numbers an operator feels.
This example routes 100 tokens to four experts with capacity 30 under two
policies. The skewed policy sends about 70% of tokens to expert 0; the balanced
policy assigns token t to expert t % 4. The skewed run reports a maximum load
of 60, 36 queued and 6 dropped. The balanced run reports a maximum load of 25,
no queue and no drops. Those are simulator outputs, and the simulator’s capacity
semantics need reading carefully before quoting them.
Run it
From the canonical repository at commit
f958967d70992d7b09d9c75000bd8c543f1644d4:
python examples/ch20/routing.py --jsonPython 3.11 or later, standard library only. No model, network, GPU or credentials. It finishes immediately.
Implementation
Each token gets one assignment. The capacity loop increments the load if it is
below capacity; otherwise it counts the token as queued and allows the load to
grow to twice capacity before counting a drop. That is why max_load can exceed
the nominal capacity.
def route_tokens(tokens: List[int], num_experts: int, capacity: int, policy: str = "top1", seed: int = 0) -> RoutingResult:
rng = random.Random(seed)
# Each token chooses an expert; skew can be simulated by bias.
assignments: List[int] = []
for t in tokens:
if policy == "skewed":
# 70% to expert 0.
if rng.random() < 0.7:
assignments.append(0)
else:
assignments.append(rng.randrange(1, num_experts))
elif policy == "balanced":
assignments.append(t % num_experts)
else:
assignments.append(rng.randrange(num_experts))
# Apply capacity.
loads = Counter()
dropped = 0
queued = 0
for expert in assignments:
if loads[expert] < capacity:
loads[expert] += 1
else:
# Queue if possible else drop.
queued += 1
if loads[expert] < capacity * 2:
loads[expert] += 1
else:
dropped += 1
load_values = [loads[e] for e in range(num_experts)]
max_load = max(load_values) if load_values else 0
mean_load = sum(load_values) / len(load_values) if load_values else 0.0
variance = sum((x - mean_load) ** 2 for x in load_values) / len(load_values) if load_values else 0.0
return RoutingResult(
policy=policy,
num_tokens=len(tokens),
num_experts=num_experts,
capacity=capacity,
max_load=max_load,
dropped=dropped,
queued=queued,
load_variance=variance,
metadata={"seed": seed},
)Expected outcome
Observed with the command above (exit 0):
{
"skewed": {
"policy": "skewed",
"num_tokens": 100,
"num_experts": 4,
"capacity": 30,
"max_load": 60,
"dropped": 6,
"queued": 36,
"load_variance": 445.25,
"metadata": {
"seed": 1
}
},
"balanced": {
"policy": "balanced",
"num_tokens": 100,
"num_experts": 4,
"capacity": 30,
"max_load": 25,
"dropped": 0,
"queued": 0,
"load_variance": 0.0,
"metadata": {
"seed": 1
}
}
}The balanced policy’s maximum load of 25 is below capacity 30. The skewed
policy’s maximum load of 60 is above capacity because this simulator counts
queued work into the load. The variance falls from 445.25 to 0.0. The
balanced case also has no queue and no drops, but it was constructed by
round-robin assignment, not by a learned router.
Mechanism and limitations
Documented: the skewed policy assigns a token to expert 0 with probability
0.7 and otherwise to a random other expert. The balanced policy assigns by
t % num_experts. The capacity loop distinguishes below-capacity, queued and
dropped states. The tests check the skewed and balanced outcomes, the drop
count, the load variance, capacity overflow and JSON serialisation.
Observed: the command exited 0 with the two result objects above. The
targeted tests.test_ch20_routing is part of the canonical suite; no independent
test count is claimed here.
Not shown: a trained router, token-level top-k routing, expert parallelism,
communication, real capacity factors, or a serving latency distribution. The
simulator’s queued counter does not model waiting time, and its max_load is
not a hard capacity violation in a real engine. A balanced assignment also does
not prove balanced work: experts can differ in cost per token.
Challenge this result
Set capacity to 10 and rerun. Predict queued and dropped for both policies,
then compare. Then change the skewed probability from 0.7 to 0.95 and explain
which summary statistic moves first. Finally, design a policy that keeps
max_load <= capacity without dropping tokens, and state what it must assume
about future arrivals or expert speed. If it cannot, say why a capacity limit
must eventually refuse or delay work.
Source and verification
Canonical source:
examples/ch20/routing.py
at commit f958967d70992d7b09d9c75000bd8c543f1644d4. Tests:
tests/test_ch20_routing.py
.
Chapter: Expert Routing and Load Balancing
.
The run above is a fresh local execution on Windows 11 with Python 3.13.15; it
is a deterministic routing simulation, not a model measurement.
Understand this example
Copy this prompt into your AI tool. No code runs here.
Apply this example
Copy this prompt into your AI tool. No code runs here.