A common agent architecture starts simply:

request
   ↓
model
   ↓
action

That simplicity is valuable.

It should be your default.

But eventually you may notice something strange.

The same model is being asked to do everything:

• classify the task,
• search documentation,
• reason about code,
• write SQL,
• review a patch,
• summarize logs,
• judge another model,
• decide whether a deployment is safe,
• and answer simple questions that did not require an expensive model in the first place.

At that point the problem may no longer be:

How do I make the model smarter?

It may be:

Why am I sending every problem to the same expert?

That is where an agent-level Mixture of Experts becomes useful.

But the phrase is easy to misunderstand.

This post is not primarily about the Mixture-of-Experts layers used inside large neural networks.

It is about something much simpler and much more directly useful when building agent systems:

task
 ↓
router
 ├──→ local model
 ├──→ frontier model
 ├──→ code specialist
 ├──→ retrieval specialist
 ├──→ deterministic tool
 ├──→ critic
 └──→ verifier

The important idea is not that more experts are automatically better.

The important idea is that different work should be routed to the mechanism most suited to doing it.

And, as with every technique in this series, that additional complexity has to earn its place.

The Search Problem This Post Solves

People building production agents eventually search for questions like:

• How do I route tasks between multiple LLMs?
• How do I use a local model and GPT/Claude together?
• How do I choose the best model for each prompt?
• Why is my agent sending easy tasks to an expensive model?
• Why does my multi-agent router choose the wrong specialist?
• How do I build a mixture of agents?
• How do I route between tools and models?
• How do I use specialist AI agents?
• Should I use multiple AI models in one application?
• How do I reduce LLM costs without reducing quality?
• Why does my expert router collapse onto one model?
• How do I know whether agent specialization actually helps?

Those look like different questions.

Architecturally, they are usually the same problem:

many possible capabilities
        ↓
one incoming task
        ↓
which capability should receive it?

That is a routing problem.

Neural MoE and Agent MoE Are Different Things

Before building anything, we need to separate two ideas that share a name.

Neural Mixture of Experts

Inside a neural network, a router may decide which learned expert layers process a token or representation:

input representation
       ↓
     router
    /  |  \
   ↓   ↓   ↓
 E1   E2   E3
    \  |  /
      ↓
   network output

The experts are parts of one learned model.

The routing happens inside the neural architecture.

Agent-Level Mixture of Experts

At the software-system level, our experts can be entirely different components:

user task
   ↓
router
   ├── local LLM
   ├── frontier LLM
   ├── code model
   ├── retrieval system
   ├── SQL specialist
   ├── deterministic calculator
   ├── browser agent
   └── verifier

These experts may differ in:

• model family,
• model size,
• prompt,
• tools,
• memory,
• cost,
• latency,
• context window,
• domain specialization,
• or even whether they are an LLM at all.

This post is about the second architecture.

The Simplest Possible Router

We do not need machine learning to begin.

Suppose our application receives four kinds of work:

calculation
coding
research
ordinary conversation

The first router could literally be ordinary Python:

from dataclasses import dataclass
from typing import Literal

ExpertName = Literal[
    "calculator",
    "code_model",
    "research_model",
    "general_model",
]


@dataclass
class RouteDecision:
    expert: ExpertName
    reason: str


def route(task: str) -> RouteDecision:
    lower = task.lower()

    if "calculate" in lower or "sum" in lower:
        return RouteDecision(
            expert="calculator",
            reason="Task appears deterministic and arithmetic.",
        )

    if "python" in lower or "bug" in lower or "code" in lower:
        return RouteDecision(
            expert="code_model",
            reason="Task appears to require code reasoning.",
        )

    if "research" in lower or "sources" in lower:
        return RouteDecision(
            expert="research_model",
            reason="Task appears to require evidence gathering.",
        )

    return RouteDecision(
        expert="general_model",
        reason="No specialist route matched.",
    )

That is already a mixture-of-experts system.

Nothing about MoE requires a neural router.

This matters because the cheapest router that works is usually the best starting point.

Why Routing Can Beat One Giant Generalist

Imagine every task is currently sent to your strongest model.

The architecture is:

all tasks
   ↓
expensive frontier model

That can be perfectly reasonable.

But suppose your workload looks like this:

55% straightforward classification
20% retrieval
15% code reasoning
 8% difficult reasoning
 2% genuinely hard edge cases

Using the most expensive model for 100% of requests may be wasteful.

A router could instead create:

request
  ↓
router
  ├─ 55% → cheap classifier/local model
  ├─ 20% → retrieval path
  ├─ 15% → code specialist
  ├─  8% → medium model
  └─  2% → frontier model

The potential gain is not only financial.

Specialists can also have:

• narrower prompts,
• fewer irrelevant tools,
• tighter schemas,
• smaller contexts,
• stronger deterministic checks,
• and more appropriate verification.

So routing may improve both efficiency and reliability.

But only if the router is good enough.

The Router Is Now Part of Your Failure Surface

Once we add routing, we have created a new subsystem that can fail.

Before routing:

model failure

After routing:

routing failure
      or
expert failure
      or
verification failure

This distinction is critical.

Suppose a coding task gets sent to the research specialist and fails.

The code expert may be excellent.

The system still fails because the router never reached it.

So we need to measure:

routing quality
separately from
expert quality

Define Experts by Capability, Not Personality

One common mistake in multi-agent systems is defining roles like:

The Architect
The Skeptic
The Genius
The Researcher
The Visionary

Those names sound interesting.

They tell the runtime almost nothing useful.

Prefer capability contracts.

For example:

from dataclasses import dataclass, field


@dataclass
class ExpertSpec:
    name: str
    capabilities: set[str]
    cost_tier: int
    latency_tier: int
    tools: set[str] = field(default_factory=set)
    notes: str = ""

Then:

experts = [
    ExpertSpec(
        name="local_classifier",
        capabilities={"classify", "route", "extract"},
        cost_tier=1,
        latency_tier=1,
    ),
    ExpertSpec(
        name="code_specialist",
        capabilities={"code", "debug", "review"},
        cost_tier=2,
        latency_tier=2,
        tools={"read_file", "search_code", "run_tests"},
    ),
    ExpertSpec(
        name="research_specialist",
        capabilities={"research", "retrieve", "synthesize"},
        cost_tier=2,
        latency_tier=2,
        tools={"search", "open_source", "extract_evidence"},
    ),
    ExpertSpec(
        name="frontier_reasoner",
        capabilities={"reason", "ambiguous", "hard_case"},
        cost_tier=4,
        latency_tier=4,
    ),
]

Now the router has something concrete to reason over.

Routing Is Classification Over Capabilities

At its simplest, routing is classification:

input task
   ↓
features
   ↓
classify required capability
   ↓
select compatible expert

For example:

"Fix the failing pytest test"
        ↓
capability = code/debug/test
        ↓
code_specialist

Or:

"What was our revenue last quarter?"
        ↓
capability = structured_data_query
        ↓
SQL / analytics expert

Or:

"What is 17.4 × 8.2?"
        ↓
capability = deterministic_math
        ↓
calculator

Notice the final example.

The best expert is not necessarily an LLM.

That is one of the most useful ideas in agent-level MoE.

Deterministic Experts Belong in the Mixture Too

A mixture of experts can contain:

LLMs
retrievers
calculators
SQL engines
static analyzers
compilers
test runners
rule engines
search systems
specialized ML models

If a deterministic system can answer a question more reliably, it should often win the route.

For example:

def preferred_expert(capability: str) -> str:
    deterministic = {
        "arithmetic": "calculator",
        "sql": "database",
        "syntax_check": "parser",
        "tests": "test_runner",
    }

    if capability in deterministic:
        return deterministic[capability]

    return "llm"

This is not less agentic.

It is better systems engineering.

A Router Should Return More Than a Name

A production route decision should be inspectable.

For example:

from dataclasses import dataclass


@dataclass
class RouteDecision:
    expert: str
    confidence: float
    required_capabilities: list[str]
    alternatives: list[str]
    reason: str

Then the runtime can distinguish:

high-confidence route
from
ambiguous route

That enables adaptive behavior.

Low Confidence Should Change Control Flow

Suppose the router returns:

RouteDecision(
    expert="code_specialist",
    confidence=0.96,
    required_capabilities=["code", "debug"],
    alternatives=[],
    reason="Explicit request to debug Python code.",
)

That is probably safe to route directly.

But:

RouteDecision(
    expert="research_specialist",
    confidence=0.52,
    required_capabilities=["research", "code"],
    alternatives=["code_specialist"],
    reason="Task requires both repository inspection and external evidence.",
)

That uncertainty should matter.

Possible responses include:

route to multiple experts
      or
ask a stronger router
      or
use a generalist
      or
split the task
      or
route sequentially

Uncertainty is not merely metadata.

It should affect the architecture.

Hard Routing

The simplest MoE architecture chooses exactly one expert:

request
  ↓
router
  ↓
expert B
  ↓
result

This is hard routing.

It is cheap and easy to understand.

Its main failure mode is obvious:

wrong route = potentially lost task

If the router chooses the wrong expert, the correct expert never gets a chance.

Top-K Routing

Instead of selecting one expert, we can select several:

request
  ↓
router
  ├→ expert B
  └→ expert D
       ↓
aggregate / verify

This is useful when:

• route confidence is low,
• two capabilities are genuinely required,
• specialists provide complementary evidence,
• or the cost of a wrong route is high.

But it also increases:

• calls,
• latency,
• aggregation complexity,
• and correlated failure risk.

Do not make k=3 your default merely because it feels safer.

Measure it.

Sequential Routing

Some tasks are not best solved by parallel experts.

They require a sequence.

For example:

research specialist
        ↓
extract evidence
        ↓
code specialist
        ↓
implement change
        ↓
verifier

This is closer to a routed workflow than classical MoE.

But the same expert-selection principle applies.

The important question is:

Which capability should receive the current state next?

That means routing can happen repeatedly during one trajectory.

Routing Is Not the Same as Multi-Agent Conversation

A mixture of experts does not require experts to talk to each other.

This is perfectly valid:

router
  ↓
expert
  ↓
verifier

No committee.

No debate.

No agent group chat.

That simplicity is often an advantage.

We should only introduce agent-to-agent communication when information genuinely has to flow between specialists.

The Local Model + Frontier Model Pattern

One of the most useful real-world MoE patterns is:

cheap/local model first
        ↓
can it solve task confidently?
   ├─ yes → verify → return
   └─ no  → frontier model

This architecture can dramatically reduce expensive calls if many tasks are easy.

A simple escalation policy might be:

def choose_model(
    complexity: float,
    uncertainty: float,
    previous_failure: bool,
) -> str:
    if previous_failure:
        return "frontier"

    if complexity > 0.75:
        return "frontier"

    if uncertainty > 0.60:
        return "frontier"

    return "local"

But remember:

model confidence
≠
correctness

The escalation should ideally use external verification too.

Verification-Driven Escalation

A stronger design is:

local expert
    ↓
verification
 ├─ PASS → return
 ├─ FAIL → escalate
 └─ UNKNOWN → escalate

Now the system does not need to trust the local model’s self-assessment.

For example:

def route_after_local(result, verification):
    if verification.status == "PASS":
        return "accept"

    if verification.status in {"FAIL", "UNKNOWN"}:
        return "frontier_model"

This connects directly to the verification architecture from the core agents series.

Cheap Experts First Is a Compute Policy

We can think of MoE routing as compute allocation.

Suppose:

local model      = 1 unit
specialist model = 3 units
frontier model   = 12 units

Then the router is effectively deciding:

How much inference compute does this task deserve?

That connects agent-level MoE to the broader theme of the advanced series:

Advanced agent architecture is often really compute allocation under uncertainty.

Cost-Aware Routing

We can explicitly include cost in route selection.

For example:

@dataclass
class CandidateRoute:
    expert: str
    predicted_success: float
    estimated_cost: float
    estimated_latency_ms: float


def utility(route: CandidateRoute) -> float:
    return (
        route.predicted_success
        - 0.03 * route.estimated_cost
        - 0.0001 * route.estimated_latency_ms
    )

Then:

chosen = max(routes, key=utility)

This is intentionally simplistic.

The important idea is that route selection can optimize more than raw quality.

Production systems care about:

• success,
• cost,
• latency,
• privacy,
• locality,
• tool access,
• context limits,
• and safety constraints.

Capability Constraints Should Come Before Preference Scores

Do not let a cheap expert win a route it cannot perform.

First filter by hard constraints:

def compatible(expert: ExpertSpec, required: set[str]) -> bool:
    return required.issubset(expert.capabilities)

Then rank compatible experts.

Architecture:

all experts
   ↓
hard capability filter
   ↓
compatible experts
   ↓
soft scoring
   ↓
selected route

Hard constraints should not be traded away by a fuzzy score.

Experts Need Sharp Boundaries

Suppose we define these experts:

code_expert
software_expert
programming_expert
engineering_expert
technical_expert

What should the router do?

These experts overlap heavily.

The router now has an unnecessarily difficult classification problem.

Prefer boundaries like:

repository_retrieval
patch_generation
test_diagnosis
architecture_review
security_review

Or, at model level:

cheap_generalist
code_specialist
long_context_researcher
frontier_reasoner

The experts should have a reason to exist.

Expert Collapse

A common MoE failure is expert collapse.

The router discovers one expert that performs reasonably well and sends nearly everything there.

You intended:

25% A
25% B
25% C
25% D

You get:

 2% A
 3% B
92% C
 3% D

This is not automatically wrong.

Perhaps C really is best.

But it is worth investigating.

Measure expert utilization:

from collections import Counter

counts = Counter(route.expert for route in route_history)

Then inspect:

• task distribution,
• expert quality,
• router calibration,
• expert overlap,
• and whether some experts are unnecessary.

Do not force equal traffic simply to make the architecture look balanced.

Redundant Experts Are Another Failure Mode

Suppose two experts consistently produce nearly identical outputs.

Then the second expert may not be providing meaningful specialization.

Measure marginal contribution.

For expert E:

system success with E
        -
system success without E

If the difference is near zero, ask why E exists.

This is a powerful ablation.

Routing Accuracy Is Not Enough

Imagine your router selects the “correct” expert 95% of the time according to labels.

That sounds excellent.

But what if the expert itself performs poorly?

We need several metrics.

routing accuracy
expert success conditional on route
overall verified success
cost
latency

The real objective is not:

correct expert label

It is:

verified task success

Routing accuracy is only an intermediate metric.

Build a Routing Confusion Matrix

Suppose the expected classes are:

code
research
data
general

Then track:

expected → selected

Example:

              selected
expected      code research data general
----------------------------------------
code           91      4     2      3
research        7     84     4      5
data            5      3    89      3
general         6      4     2     88

Immediately we can see:

research → code

is a recurring confusion.

That gives us something concrete to fix.

Why the Router Makes Mistakes

Common causes include:

1. Overlapping expert descriptions

code expert: handles technical programming tasks
research expert: handles technical investigation

Many tasks match both.

2. Missing task context

The router receives:

"Fix this"

instead of repository state, failure type, and user objective.

3. Too many experts

Every added route increases the classification space.

4. Bad labels

Your benchmark may disagree with what actually produces the best verified outcome.

5. Stale expert capabilities

The router thinks an expert supports a tool or context size it no longer supports.

6. Cost dominates too strongly

The optimizer repeatedly selects a cheap but inadequate expert.

Route by State, Not Just by User Prompt

Agent routing becomes much more powerful when the router sees runtime state.

For example:

@dataclass
class AgentState:
    task: str
    step: int
    failures: list[str]
    evidence: list[str]
    last_expert: str | None
    verification_status: str | None

Then routing can change during the trajectory.

Example:

initial task
   ↓
local code model
   ↓
tests still fail
   ↓
debug specialist
   ↓
verification unknown
   ↓
frontier reasoner

This is much more interesting than one-time prompt classification.

Failure-Aware Routing

Repeated failure is useful routing evidence.

Suppose the same expert has failed twice.

Do not simply send the task back unchanged.

def should_escalate(state: AgentState) -> bool:
    return len(state.failures) >= 2

Then:

same expert fails repeatedly
         ↓
change expert / model / strategy

This connects routing to the loop-control ideas from the earlier series.

A Full Minimal Agent-Level MoE Runtime

Here is a small provider-independent skeleton.

from dataclasses import dataclass, field
from typing import Callable, Any


@dataclass
class Expert:
    name: str
    capabilities: set[str]
    run: Callable[[dict], Any]
    cost_tier: int = 1


@dataclass
class RouteDecision:
    expert: str
    confidence: float
    alternatives: list[str] = field(default_factory=list)
    reason: str = ""


@dataclass
class ExpertResult:
    expert: str
    output: Any
    verified: bool | None = None


class ExpertRegistry:
    def __init__(self):
        self._experts: dict[str, Expert] = {}

    def register(self, expert: Expert) -> None:
        self._experts[expert.name] = expert

    def get(self, name: str) -> Expert:
        return self._experts[name]

    def compatible(self, capabilities: set[str]) -> list[Expert]:
        return [
            expert
            for expert in self._experts.values()
            if capabilities.issubset(expert.capabilities)
        ]


class AgentMoE:
    def __init__(self, registry, router, verifier):
        self.registry = registry
        self.router = router
        self.verifier = verifier

    def run(self, task: dict) -> ExpertResult:
        required = set(task.get("required_capabilities", []))
        candidates = self.registry.compatible(required)

        if not candidates:
            raise RuntimeError(
                f"No expert supports capabilities: {sorted(required)}"
            )

        decision = self.router(task, candidates)
        expert = self.registry.get(decision.expert)

        output = expert.run(task)
        verification = self.verifier(task, output)

        if verification == "PASS":
            return ExpertResult(
                expert=expert.name,
                output=output,
                verified=True,
            )

        if verification in {"FAIL", "UNKNOWN"}:
            for alternative_name in decision.alternatives:
                alternative = self.registry.get(alternative_name)
                output = alternative.run(task)
                verification = self.verifier(task, output)

                if verification == "PASS":
                    return ExpertResult(
                        expert=alternative.name,
                        output=output,
                        verified=True,
                    )

        return ExpertResult(
            expert=expert.name,
            output=output,
            verified=False,
        )

This gives us the essential architecture:

capability filter
      ↓
router
      ↓
expert
      ↓
verifier
      ↓
fallback / escalation

Routing Can Be Deterministic

Do not immediately train a router.

A rule-based router can be excellent when the domains are clear.

Example:

def deterministic_router(task, candidates):
    capabilities = set(task["required_capabilities"])

    if "arithmetic" in capabilities:
        target = "calculator"
    elif "code" in capabilities:
        target = "code_specialist"
    elif "research" in capabilities:
        target = "research_specialist"
    else:
        target = "general_model"

    alternatives = [
        expert.name
        for expert in candidates
        if expert.name != target
    ]

    return RouteDecision(
        expert=target,
        confidence=1.0,
        alternatives=alternatives,
        reason="Deterministic capability rule.",
    )

If this solves 95% of your routing problem, keep it.

LLM Router

When task boundaries are fuzzy, an LLM can classify the request.

But the router itself should have a constrained output schema.

For example:

{
  "required_capabilities": ["code", "debug"],
  "selected_expert": "code_specialist",
  "confidence": 0.83,
  "alternatives": ["frontier_reasoner"]
}

Then validate that:

• the expert exists,
• the expert supports the capabilities,
• confidence is within bounds,
• alternatives exist,
• the route is authorized.

The router should not be allowed to invent experts.

Learned Router

Eventually you may have enough routing evidence to train a small classifier.

Inputs might include:

task embedding
repository type
tool requirements
context length
failure history
cost budget
latency budget

Output:

P(expert | task, state)

This can be valuable at high volume.

But it creates a new model that must itself be evaluated, versioned, calibrated, and monitored.

Do not build it because “MoE should have a learned router.”

Build it because the routing dataset shows deterministic and prompt-based routing have become the bottleneck.

Router Training Data Should Come From Outcomes

A subtle point:

The best training label is not necessarily:

which expert sounds appropriate?

It is closer to:

which expert produced the best verified outcome
for this task under the relevant cost/latency constraints?

That means routing data should ideally contain:

@dataclass
class RoutingEvidence:
    task_id: str
    expert: str
    verified_success: bool
    cost: float
    latency_ms: float
    failure_type: str | None

Now router improvement becomes evidence-driven.

Expert Selection Can Be Contextual Bandit-Like

Once the system gathers outcome evidence, routing begins to resemble a contextual bandit problem:

context
  ↓
choose expert
  ↓
observe reward/outcome
  ↓
update routing policy

That does not mean you should immediately deploy online learning.

But it gives us a useful conceptual model:

routing is a decision under uncertainty
with observable downstream consequences

This becomes important later when we discuss adaptive agents.

Specialist Judges Are Experts Too

Suppose we generate code with one model.

We might route the result to specialists:

patch
 ↓
syntax verifier
 ↓
test runner
 ↓
security reviewer
 ↓
performance reviewer

These are not necessarily candidate generators.

They are expert evaluators.

So an agent MoE can exist on both sides:

generation experts
and
evaluation experts

This is especially useful when failure classes differ significantly.

But Do Not Build a Reviewer Army Without Evidence

Imagine:

security critic
style critic
architecture critic
correctness critic
performance critic
maintainability critic
simplicity critic

That looks thorough.

It may simply multiply cost.

Ask:

What additional verified defects does each reviewer find?

Measure marginal defect discovery.

If two reviewers find the same issues almost every time, one may be redundant.

Expert Diversity Should Be Functional

Useful diversity comes from different capabilities or evidence channels.

For example:

LLM reviewer
static analyzer
test runner
security scanner
runtime benchmark

Those are genuinely different experts.

Less useful:

same model + "be a reviewer"
same model + "be a critic"
same model + "be skeptical"

Prompt-role diversity can help.

It should not be confused with independent evidence.

Agent MoE for Coding Software

Coding agents are a natural application.

A useful mixture might look like:

engineering task
      ↓
router
      ├→ repository retrieval
      ├→ code generator
      ├→ test diagnosis
      ├→ architecture reviewer
      ├→ security reviewer
      └→ frontier escalation

Example route:

"Why is CI failing?"
      ↓
CI/log specialist
      ↓
identify failing test
      ↓
code specialist
      ↓
patch
      ↓
test verifier

The application is not “several agents talking.”

It is specialized engineering capabilities connected by explicit routing.

Agent MoE for Research Software

A research system may have:

query classifier
web retrieval specialist
paper retrieval specialist
source-quality evaluator
claim extractor
synthesis model
citation verifier

A route might be:

question
 ↓
requires current evidence?
 ├─ no → internal/general model
 └─ yes → retrieval specialist
             ↓
          evidence
             ↓
         synthesis
             ↓
       citation verifier

Again, specialization is driven by the work.

Agent MoE for Customer Support

A support architecture might route by case type:

incoming case
   ↓
classifier
   ├→ billing
   ├→ shipping
   ├→ technical support
   ├→ account access
   └→ human escalation

Each route can have different:

• tools,
• permissions,
• policies,
• verification,
• and risk limits.

This is a strong reason to route: not every specialist should have access to every action.

Agent MoE for Data and Analytics

A data agent might use:

question
 ↓
router
 ├→ SQL generator
 ├→ dataframe analyst
 ├→ statistical model
 ├→ chart generator
 └→ narrative summarizer

Important distinction:

"calculate revenue by quarter"

should probably route toward deterministic data execution.

Whereas:

"explain why revenue changed"

may require synthesis after the computation.

One request can therefore cross multiple experts sequentially.

Agent MoE for DevOps and Incident Response

An incident system could contain:

log specialist
metrics specialist
configuration specialist
network specialist
deployment specialist
rollback verifier

A route might depend on evidence:

latency spike
   ↓
metrics specialist
   ↓
DB saturation detected
   ↓
database specialist
   ↓
remediation candidate
   ↓
verification

This is state-dependent expert routing.

Agent MoE for Browser Automation

A browser system may divide work between:

page-state parser
navigation planner
form specialist
extraction specialist
visual fallback model

Most pages may be handled cheaply from DOM structure.

Only difficult visual states may need the expensive vision model.

That gives us:

DOM-first
   ↓
verification
   ↓
vision escalation only when necessary

A classic adaptive-compute MoE pattern.

Application Matrix

The important pattern is:

route based on task/state
verify using the environment

The Router Should Not Be the Final Judge

Suppose the router says:

expert = code_specialist
confidence = 0.98

That does not prove the code specialist will succeed.

The architecture still needs:

route
 ↓
execute
 ↓
verify

Routing confidence is about routing.

It is not task-success confidence.

Expert Confidence Is Also Not Enough

The expert may say:

"I am 95% confident the issue is fixed."

The test suite may say:

FAIL

The test suite wins.

The evidence hierarchy remains:

environment evidence
        >
model confidence

Fallback Should Be Explicit

Production routing needs a fallback policy.

For example:

route expert A
    ↓
verification FAIL
    ↓
expert B
    ↓
verification UNKNOWN
    ↓
frontier generalist
    ↓
verification

Avoid unlimited fallback chains.

Set budgets:

MAX_EXPERT_ATTEMPTS = 3
MAX_TOTAL_COST = 2.50
MAX_LATENCY_SECONDS = 30

The system must eventually stop.

Track Why Escalation Happened

Do not log only:

model = frontier

Log:

initial_expert = local_code
reason_for_escalation = verification_failed
failed_check = test_user_creation
frontier_expert = frontier_code

This makes expensive routes explainable.

It also lets you improve the cheap path later.

Routing Telemetry

At minimum, record:

@dataclass
class RouteTrace:
    task_id: str
    selected_expert: str
    confidence: float
    alternatives: list[str]
    route_reason: str
    verified_success: bool | None
    cost: float
    latency_ms: float
    escalated: bool
    escalation_reason: str | None

Now you can ask:

• Which expert gets the most traffic?
• Which expert has the best verified-success rate?
• Which routes fail most often?
• Where does escalation happen?
• Which expert is expensive but rarely necessary?
• Which specialist contributes unique wins?

Metrics That Actually Matter

Useful metrics include:

Router metrics

routing accuracy
routing calibration
route entropy
expert utilization

Expert metrics

verified success by expert
failure type by expert
latency by expert
cost by expert

System metrics

overall verified success
cost per verified success
p50/p95 latency
escalation rate
fallback success rate

Specialization metrics

marginal success contribution
unique defect discovery
expert redundancy

These tell us whether MoE is actually doing useful work.

Route Entropy

One simple diagnostic is route entropy.

If every request goes to the same expert, route entropy is low.

That may mean:

• expert collapse,
• the router is biased,
• or one expert genuinely dominates.

If every request is distributed almost uniformly, route entropy is high.

That may mean:

• tasks are genuinely diverse,
• or the router is uncertain/noisy.

The metric is diagnostic, not an objective.

Do not optimize entropy for its own sake.

When More Experts Make the System Worse

Adding experts can reduce reliability because it increases:

routing ambiguity
configuration complexity
observability burden
fallback complexity
benchmarking surface

Suppose success is:

one model = 89%
3 experts = 92%
8 experts = 91%
16 experts = 88%

The sophisticated architecture lost.

More specialists did not create more specialization.

They created a harder routing problem.

Do Not Add an Expert Without a Failure It Owns

A useful rule:

Every expert should own a recognizable failure class or capability boundary.

For example:

Expert: test_diagnosis
Failure it addresses:
General model changes code before understanding the failing test.

Or:

Expert: long_context_retrieval
Failure it addresses:
General model loses relevant repository context in large codebases.

Or:

Expert: frontier_escalation
Failure it addresses:
Local model fails on the small subset of genuinely difficult tasks.

If you cannot write that sentence, you may not need the expert.

Ablate Experts One at a Time

A simple experiment:

full system
full system - expert A
full system - expert B
full system - expert C

Measure:

verified success
cost
latency
failure coverage

If removing an expert changes nothing, that is strong evidence it may be unnecessary.

Compare Against the One-Model Baseline

The most important experiment is still:

one strong model
vs
routed mixture

A useful table:

These numbers are illustrative.

The question is not:

Which architecture sounds advanced?

It is:

Which architecture produces the best verified outcome
under our actual constraints?

Run an Oracle Routing Experiment

One particularly useful experiment is oracle routing.

For every benchmark task, run all candidate experts offline.

Then determine:

which expert actually produced the best verified result?

Now you can estimate an upper bound:

oracle expert selection

Compare:

router-selected success
vs
oracle-selected success

If oracle routing is only slightly better than one model, specialization itself has limited value.

If oracle routing is dramatically better but your router performs poorly, the bottleneck is routing.

This separates two questions beautifully:

Does specialization help?

and

Can we route to it reliably?

Router Regret

We can define a simple routing regret concept:

oracle outcome
    -
selected-route outcome

For score-based tasks:

routing_regret = oracle_score - selected_score

For binary verified success, track how often:

a successful expert existed
but the router chose a failing expert

That is one of the most useful routing metrics you can collect.

Hard Cases Should Generate Training Evidence

When routing fails:

store the task
store the selected route
store candidate expert outcomes
store verification
store cost/latency

Now routing errors become future evidence.

That is how a static router can eventually become an adaptive one.

But that is a later mechanism.

Memory of outcomes is not yet learning.

Do Not Confuse Memory With Adaptive Routing

A system may retrieve:

"Last time this kind of error went to the DB specialist."

That is memory.

If the routing policy itself changes because repeated outcomes demonstrate that the DB specialist works better, that is learning/adaptation.

We will return to that distinction later in this series.

MoE and Search Can Be Combined

The architectures in this series are composable.

For example:

MCTS node
   ↓
router
   ├→ code expert
   ├→ research expert
   └→ verifier

Different branches could use different experts.

Or:

router
  ↓
expert
  ↓
Tree of Thoughts inside expert

But composition multiplies cost and failure surfaces quickly.

Build one mechanism at a time.

Benchmark every escalation.

MoE and Self-Consistency Can Be Combined

Instead of sampling one model five times:

same model × 5

we might sample specialists:

code model
frontier generalist
local model
static analyzer

That can produce more functional diversity.

But aggregation still needs strong evidence.

Expert disagreement does not become truth through voting.

MoE and Verification Are Natural Partners

A powerful architecture is:

cheap specialist
      ↓
verification
  ├─ PASS → finish
  └─ FAIL/UNKNOWN
          ↓
      stronger expert
          ↓
      verification

This creates an evidence-driven compute cascade.

Instead of predicting difficulty perfectly in advance, the runtime uses failed verification as evidence that more compute is required.

That is often easier to engineer than a perfect complexity classifier.

Safety and Permissions Belong in Expert Definitions

Experts should not all have the same permissions.

Example:

@dataclass
class ExpertPolicy:
    expert: str
    allowed_tools: set[str]
    can_write: bool
    can_deploy: bool
    can_access_secrets: bool

A research expert might have:

read/search only

A deployment expert may have:

production actions

Routing therefore affects authorization.

The runtime must validate both:

Is this expert suitable?

and

Is this expert allowed to perform this action?

Never Let the Router Invent Authority

Even if the router chooses:

deployment_expert

that does not automatically grant production access.

Authorization must be independent.

Architecture:

router proposes expert
        ↓
authorization layer
        ↓
allowed capability subset
        ↓
execution

The model does not get to increase its own permissions by changing the route.

Production Debugging Checklist

If your routed agent system performs badly, inspect it in this order.

1. Does specialization actually help?

Run oracle routing.

If every expert performs similarly, the architecture may not need specialists.

2. Is the router choosing the wrong expert?

Inspect confusion matrix and routing regret.

3. Are expert boundaries overlapping?

Merge or sharpen ambiguous experts.

4. Is one expert receiving almost everything?

Inspect expert collapse, but do not force balance without evidence.

5. Are cheap experts failing and escalating constantly?

The cheap tier may be too weak or the route threshold too permissive.

6. Are expensive experts called unnecessarily?

Tighten escalation and deterministic routes.

7. Do specialists produce unique wins?

Ablate each expert.

8. Is the router using stale capability metadata?

Version expert specs.

9. Are verification failures being mistaken for routing failures?

Separate subsystem telemetry.

10. Does the whole mixture beat one strong model?

If not, simplify.

A Controlled Experiment

Suppose we are building a coding assistant.

Benchmark these architectures:

A: frontier model for every task
B: local model for every task
C: rule router → local/code/frontier
D: learned/LLM router → specialists
E: C + verification-driven escalation

Record:

verified task success
routing regret
oracle routing success
local-model utilization
frontier escalation rate
model calls
latency
cost

Then introduce failure buckets:

simple Q&A
repository retrieval
bug diagnosis
patch generation
test repair
architecture review

Now you can see where specialization actually helps.

The Result We Want

The goal is not:

more agents

It is closer to:

cheap mechanism for easy work
specialist mechanism for specialized work
frontier model for genuinely hard work
deterministic tools where deterministic tools are stronger
verification after every consequential outcome

That is a much more useful interpretation of mixture of experts.

A Practical Escalation Ladder

Start here:

one model

If cost is the problem:

cheap model
   ↓
verification-driven frontier escalation

If domain failures are concentrated:

router
 ├→ domain specialist A
 ├→ domain specialist B
 └→ generalist

If route uncertainty matters:

top-k experts
   ↓
strong aggregation / verification

If routing itself becomes the bottleneck:

learned router

Only add the next stage when the previous architecture produces evidence that it is needed.

The Core Principle

The essential insight is simple:

A mixture of experts is not a collection of agents. It is a policy for assigning work to the capability most likely to solve it under your cost, latency, safety and verification constraints.

That policy can be:

• deterministic,
• model-based,
• learned,
• state-dependent,
• failure-aware,
• cost-aware,
• or verification-driven.

The architecture becomes useful when specialization creates real differences in outcome.

If every expert behaves the same, you do not have a meaningful mixture.

If routing adds more failure than specialization removes, simplify.

If one cheap expert solves almost everything, let it.

And if only a small fraction of requests truly require frontier-level inference, do not make every request pay for it.

Where We Are in the Series

We now have:

00  When do advanced agent architectures help?
 ↓
01  Chain of thought as intermediate computation
 ↓
02  Self-consistency and disagreement
 ↓
03  Tree of Thoughts
 ↓
04  Monte Carlo Tree Search
 ↓
05  Agent-level Mixture of Experts

So far we have explored two major advanced-agent dimensions:

search
and
routing / specialization

The next step is to combine specialized responsibilities into a deliberate control architecture.

That means moving beyond one router selecting one expert and asking:

What if planning, execution, criticism and verification should be separate roles with explicit contracts?

That is the subject of the next post:

Advanced Agents From First Principles 06: Does One Agent Plan, Execute and Judge Its Own Work? Build a Planner–Executor–Critic Architecture.