Part 6 — Put Intelligence Into the Process

A claim awaiting a check

Constructed scene. A patch sits open with one review comment unresolved: a retry test that may or may not cover the new branch. Three things could happen next. A model could propose a fix. A deterministic check could run the test suite. A person could be asked whether the branch matters at all. Each costs something different, fails differently, and answers a different question. “Try a stronger model” is not on that list until something has established that generation is the missing operation.

Chapter 27 ended with a policy that had to earn promotion and did not. Every experiment in Part 5, though, spent model calls freely in order to ask its question. This chapter turns the same discipline on the spending itself: the next model call should have a reason, and the model stops being the implicit controller of the loop.

Which operation is needed before choosing a model?

Three decisions that usually get merged

“Route to the right model” hides three separate decisions:

Keeping them apart changes what failure means. Choosing a cheaper model cannot repair choosing the wrong operation. A failed model call is not automatically a reason to buy a stronger one; it may be a reason to run a check, or to stop climbing and ask a person. And the evidence for one decision does not transfer to another. Stage 29B, the chapter’s measured center, tests escalation. It says nothing directly about whether an operation selector should be deterministic — which is the question the unrun experiment exists to ask.

The literature has names for two of the halves. Modular architectures that route among language models, knowledge sources, and discrete reasoners treat which component answers as a decision separate from generation (Karpas et al., 2022). That supports treating operation choice as its own layer, though a modular diagram does not demonstrate a fixed-priority scheduler, and the mapping onto CodeAI is the book’s. Learned routers attack the neighboring decision: given that a model will be called, which one, trading quality against cost with routers trained on preference data (Ong et al., 2024). Neither paper validates CodeAI’s ladder.

The operation seam

Here is the whole current policy, from CodeAI source:

def decide_next_step(query: SchedulerInput) -> SchedulerDecision:
    if query.process_budget_exhausted:
        return SchedulerDecision(Operation.STOP, "process budget exhausted", POLICY_VERSION)
    if query.has_required_verification:
        return SchedulerDecision(Operation.CHECK, "required deterministic verification exists", POLICY_VERSION)
    if query.requests_independent_proposals and not query.model_budget_exhausted:
        return SchedulerDecision(Operation.CALL, "task requests independent proposals", POLICY_VERSION)
    if query.requires_human_authority_for_next_effect:
        reason = "next effect requires human authority"
        if query.model_budget_exhausted and query.requests_independent_proposals:
            reason += "; model budget blocks proposals"
        return SchedulerDecision(Operation.ASK_HUMAN, reason, POLICY_VERSION)
    if query.model_budget_exhausted and query.requests_independent_proposals:
        return SchedulerDecision(Operation.STOP, "model budget blocks proposals", POLICY_VERSION)
    return SchedulerDecision(Operation.STOP, "no epistemic operation required", POLICY_VERSION)

¹

No model, no I/O, no ledger write. Policy epistemic-v2 orders five caller-supplied flags, and its most important design choice is the one that is easiest to miss: two different budgets. Model-budget exhaustion blocks CALL and nothing else. Process-budget exhaustion stops everything. Collapse them into one “budget exhausted” flag and a system that has spent its cognition allowance can no longer run the check it still owes.

The policy as a decision tree, in precedence order — operation first, model choice later and elsewhere:

    flowchart TD
    IN["scheduler input<br/><i>five caller-supplied flags</i>"] --> P{"process budget<br/>exhausted?"}
    P -->|"yes"| S1["STOP<br/><i>dominates everything</i>"]
    P -->|"no"| C{"required verification<br/>exists?"}
    C -->|"yes"| CK["CHECK<br/><i>survives a spent model budget</i>"]
    C -->|"no"| R{"proposals requested<br/>+ model budget left?"}
    R -->|"yes"| CL["CALL"]
    R -->|"no"| H{"next effect needs<br/>human authority?"}
    H -->|"yes"| AH["ASK_HUMAN"]
    H -->|"no"| M{"model budget<br/>blocks proposals?"}
    M -->|"yes"| S2["STOP<br/><i>model budget blocks proposals</i>"]
    M -->|"no"| S3["STOP<br/><i>no epistemic operation required</i>"]
    AC["ACTION<br/><i>in the enum; no input selects it</i>"]
    style AC stroke-dasharray: 4 4
  

Constructed calls against that function show the precedence, executed as teaching code rather than as a stage: ²

decide_next_step(SchedulerInput(
    has_required_verification=True,
    requests_independent_proposals=True,
    requires_human_authority_for_next_effect=True,
    process_budget_exhausted=True,
    model_budget_exhausted=True,
)).operation  # Operation.STOP: process stop dominates the other four flags

decide_next_step(SchedulerInput(
    has_required_verification=True, model_budget_exhausted=True,
)).operation  # Operation.CHECK: the check survives the spent model budget

Three boundaries are part of the design, not omissions to smooth over.

• ACTION is unreachable. The enum contains it; no input selects it. Effects are authorized downstream under explicit grants (Chapters 20 and 22). A seam path to ACTION would complete the enum cosmetically while bypassing that authority story. CALL and CHECK schedule work; neither authorizes an effect. ³
• The flags are not facts. The docstring says it directly: flags are caller-supplied state, not inferred facts or resource accounting. Whatever computes has_required_verification is outside this function and outside its guarantees. ²
• Decisions are not recorded. Runtime.decide_next is a thin passthrough; a decision reaches the ledger only if a caller persists it. ⁴

The historical scheduler matrix needs its version stamp attached. It ran all sixteen combinations of the older four-flag vocabulary against policy epistemic-v1: 16/16 parity between the code and a data-table interpreter, CALL/CHECK/ASK_HUMAN/STOP reachable, ACTION dead, and the same simultaneity precedence. The committed policy has since become the five-flag epistemic-v2 with split budgets. Legacy spellings still migrate, but no matrix covers the split-budget rows. The later change to scheduler.py between 7a0d43b and a1b562a is a dict-literal reformat with no behavioral change. ^{5 6}

That demo also settled a smaller design question. A five-row declarative table reproduced the if-chain exactly, but it duplicated the chain without fixing the real gaps — unpersisted decisions and precomputed flags. At this size, the book keeps the small function and drops the policy DSL. ⁵

The measured center: Stage 29B

The scheduler answers which operation. Stage 29B asks the question one level down: once the operation is “get an answer,” how should the process climb from cheap to expensive, and when should it stop climbing?

The workload. Forty synthetic cache-TTL extraction items with frozen gold: ten structured (CDN_TTL_SECONDS=86400), twelve prose, twelve with distractor durations, six with no single correct TTL.

The arms. The ladder climbs rule → free model → cheap model → strong model → person. Top-first sends every item to the strong model, then to a person.

The acceptance check. Every candidate passes through the same deterministic checker, blind to gold: valid JSON, a verbatim quote from the input, exactly one duration expression in that quote, and seconds equal to that duration.

The adoption rule, frozen before the run. The ladder is adopted only if all three hold at a $5-per-person-asked scenario: cheaper per accepted outcome; correct acceptances at least top-first’s minus one; accepted-but-wrong outcomes no more than top-first’s.

Calls went to named providers at declared prices, each arm-item pair in its own process, with every response body, receipt, and event preserved. ⁷

How the ladder climbs

run_ladder records every transition. A rung is entered with a reason, declines with one of five reason kinds, and the run ends resolved, with a person, or exhausted:

ladder.rung_entered   rung, reason
ladder.rung_declined  rung, reason ∈ { rule_declined:…, call_not_succeeded:…,
                                       output_unavailable:…, check_failed:…,
                                       check_error:… }
ladder.resolved | ladder.asked_human | ladder.exhausted

⁸

Two real traces from the preserved ledger show what that buys. A05 climbed one rung:

ladder.rung_entered   rule      "first rung"
ladder.rung_declined  rule      "rule_declined:0 ttl lines"
ladder.rung_entered   zen-free  "escalated from rule: rule_declined:0 ttl lines"
ladder.resolved       zen-free

T07 climbed all the way, and its decline reasons never changed:

rule_declined:0 ttl lines
check_failed:quote_has_0_durations     (free)
check_failed:quote_has_0_durations     (cheap)
check_failed:quote_has_0_durations     (strong)
→ ladder.asked_human

⁷

The router’s decision path contains no model call: run_ladder is 61 lines, _check_decline eight. ⁷

The result

Recomputed from the frozen analysis rows, matching the bundle’s tables: ⁷

The preregistered output is recorded as ladder_justified: false, with the verdict label “top-rung-first wins on this workload.” That label needs unfolding, because either half of it read alone is wrong. The ladder met two conditions: it was cheaper per accepted outcome, and it produced one more correct acceptance. It failed the third: it produced one accepted-but-wrong outcome where top-first produced none. The rule required no increase in wrong acceptances, so the ladder was not adopted and top-first remains the default for this workload under this rule. Not “the strong model is better.” Not “cheap-first lost.” One clause fired, and that clause was written to protect exactly what it protected. ⁷

The cost column rests on a scenario, not a wage. Executed with the frozen figures:

cost = model_spend + people_asked * PERSON_COST   # PERSON_COST = $5, a scenario
cost_per_accepted = cost / accepted
# ladder:    (0.0056 + 7 * 5) / 33 = 1.061
# top-first: (0.0317 + 9 * 5) / 31 = 1.453

With five dollars per person, human escalations dominate both totals. The preserved sensitivity runs at $0 (per-accepted costs of fractions of a cent) and $25 ($5.30 against $7.26); the decision is the same at both, and it is also unchanged when top-first’s unknown-spend call is priced in. The ordering is stable across those scenarios; the absolute numbers are scenario artifacts. ⁷

A05, and the accident that avoided it

The input reads: in staging the cache TTL is one minute; in production it is ten minutes. There is no single correct answer, and the gold says so. The free rung answered 600 seconds, quoting “10 minutes.” Valid JSON, verbatim quote, exactly one duration, seconds matching the quote: the checker passed it, and the acceptance was wrong.

This is Chapter 21’s adequacy lesson in a new setting. The checker verifies grounding. It was never built to verify that the question has one answer, and it did exactly what it was built to do. The checker is not tuned afterward. A05 is evidence of the checker’s limit, and repairing it post hoc would erase the measurement. ⁷

Top-first did not make the same mistake, and the preserved bytes show why. Its strong-model call finished for length with 2,047 of its 2,048 completion tokens spent on reasoning and empty content. The reasoning is preserved: the model notices the two durations, asks itself “Maybe answer should be null because not a single?”, circles the schema, and is cut off partway through drafting a null answer. The adapter classified the call as empty_output, the call failed, and the item went to a person.

So top-first’s zero came from a token limit reached mid-deliberation, not from a judgment it delivered. The reasoning suggests the model was heading toward null. It never produced an answer the checker could see, and a slightly larger token budget could have produced either outcome. What the run shows is the mechanism of this one avoidance, not better judgment by the strong model. ⁷

The checker has a mirror-image limit. T07 (gold 7,200 seconds) and T10 (gold 5,400) had correct answers the quote checker could not verify: zero duration expressions found in one quote, two in the other, on every model rung in both arms. Both items went to people after the paid rungs failed them too. A wrong answer passed; correct answers were rejected. What one deterministic checker can verify is not the same as what is true. ⁷

The rungs that earned nothing

Resolution in the ladder arm: rule 10 of 10 attempts, free model 23 of 30, cheap model 0 of 7, strong model 0 of 7, a person for the remaining 7. The arithmetic closes on its own: 10 + 23 + 7 = 40 items, and 30 + 7 + 7 = 44 model calls, which corroborates the per-item rows independently of the report’s prose. ⁷

The measured statement stops there. It does not show that stronger models cannot solve such cases, only that on this seven-item remainder the two paid rungs produced no accepted resolution. The recorded reasons suggest why: quote_has_0_durations, quote_has_2_durations, and items with no value to verify — reasons rooted in the input and the checker, which a stronger model cannot change. That motivates a decline-reason-aware policy, routing by why a rung declined rather than escalating every decline through more capacity. It is a hypothesis with a clear next experiment, not a result. ⁷

The rule rung carries the chapter’s plainest systems point. On the ten structured items, the rule resolved all ten with no model call. Top-first’s strong model got nine right; on S10 it answered with the bare digits 86400 without their key, so the quote contained no duration expression and the item went to a person. That is one item against one model, and the claim stays that size: when the operation is already mechanically specified, a model call adds a way to fail that the rule does not have, without adding anything the rule was missing. ⁷

Cost is not the only axis. The ladder spent less on models across more calls, took more summed model time (417 s against 315 s, dominated by the free rung’s reasoning), escalated far more often between rungs (51 transitions against 9), and asked people less often (7 against 9). Reducing model spend did not reduce process time. And the scope fence holds on every sentence above: one synthetic workload, one checker, one ladder configuration, one model set, one pricing period, one run per arm, no variance estimate. ⁷

The spend the ledger could not see

The A05 top-first call returned HTTP 200 with 92 input and 2,048 output tokens reported, and no text. CodeAI’s adapter raised on the missing text, and its failure path then recorded usage as unavailable and spend as unknown, discarding the usage it had been handed. The projection honestly carried one unknown-spend attempt. The independent verifier, recomputing from the preserved bytes at declared prices, found $0.0083 — the most expensive single call in the run — and failed the spend claim. The bundle keeps both: decision tables use known spend with the unknown call flagged beside them. ⁷

The cause was an ordering bug inside one function: usage was parsed after text extraction, so an extraction failure skipped it. The repair, made while this chapter was drafted and now in CodeAI a1b562a, parses usage first and carries already-parsed tokens into the failure result. Reduced from the source, with the failure path’s result fields shown as assignments:

raw_usage = parsed.get("usage") if isinstance(parsed.get("usage"), dict) else None
in_tokens, in_reported = _extract_usage(raw_usage, "prompt_tokens")
out_tokens, out_reported = _extract_usage(raw_usage, "completion_tokens")
text = _chat_text(parsed)        # may raise; usage is already in hand
...
# failure path
usage_source = "measured" if (in_reported or out_reported) else "unavailable"
cost_usd = estimate_cost_usd(self.model, in_tokens, out_tokens)
cost_source = "estimated" if cost_usd is not None else "unknown"

⁹

Three regression tests pin the boundary. The preserved A05 shape (empty content, 92/2,048 usage, a model absent from the pricing table) keeps measured tokens with unknown cost — the $0.0083 belongs to the experiment’s declared prices, not to the adapter’s table. A priced-model variant estimates cost on the failure path. A failure with no usage stays unavailable. The historical bundle is untouched and the experiment was not rerun, so the tables above still show what the run recorded. The sibling adapters share the old ordering and are named, not silently repaired. ¹⁰

One adjacent defect stays named. estimate_cost_usd matches model IDs by prefix in table order, so gpt-4o-mini is billed at gpt-4o rates, about seventeen times the intended input price. The one-line fix changes billing semantics under a pricing version that existing tests and the router design pin, so it needs its own versioned migration rather than a quiet edit here. ^{11 12}

The challenger that has not run

The book’s second standing bet is that the process’s routing decision should contain no model call. That bet now has a serious opponent. A deployed system routes in real time between fast and reasoning models using conversation type, complexity, tool needs, and explicit intent, and its router is continuously trained on signals such as users switching models, preference rates, and measured correctness (OpenAI, 2025). That establishes learned routing as a credible production architecture for choosing which model. It does not settle whether operation selection should contain a model call, which is the narrower bet here.

CodeAI’s answer is an experiment design, not a result. It asks whether a model-based router chooses the next operation better than the deterministic policy when both see the same explicit state. Its falsifier says the deterministic bet must be revised if the model-based router is at least 15 percentage points better at operation selection overall, with no stratum worse by more than 5 points; causes no more catastrophic misroutes and no new catastrophe classes; keeps median decision flip rate at or below 10% (p95 at or below 20%); refuses at most 5% of decisions; and produces reasons at least 90% non-vacuous, with auditability not materially worse. The binding runs the other way too: a deterministic router that is systematically worse is a real failure, because reproducibility alone wins nothing. A second, exploratory track separates where intelligence might pay — interpreting state versus choosing policy — and does not feed the falsifier. The design even states its own power limit: with 24 structured and 24–32 narrative cases, a small correctness difference would be suggestive at best. ¹³

Its status needs to be classified precisely, because several different things are true at once:

^{14 15 13}

So there is no router accuracy, no catastrophe count, no cost, no flip rate — nothing to report except the design and its falsifier. That is a stronger position than it sounds: the book’s architectural preference is on the table with the exact conditions under which it falls, and nothing in this chapter or the next depends on how the experiment would come out.

Two naming rules hold throughout. The model arm is model-based, never “learned”: nothing was trained on routing data. And in the exploratory track, a deterministic state compiler must answer UNKNOWN where it cannot extract a fact, never guess. Deterministic is not omniscient, and the experiment compares honest boundaries. ¹³

Checking it without trusting it

The ladder’s verifier imports neither CodeAI nor the producer. It restates the rule, the quote parser, and the check, then re-derives rung order, verdicts, decline reasons, spend from usage and declared prices, receipts, both controls, scoring, and the decision from preserved bytes. Result: 13 of 14 semantic claims pass. The one failure is the spend under-derivation described above. Five seeded corruptions — a flipped verdict, understated spend, a skipped rung, inflated acceptances, a dropped receipt — each produced problems beyond that baseline failure. A fresh process reprojected all four ledgers identically. The verifier was re-run during drafting with the same 13/14 result. ⁷

The controls also behaved. A scripted wrong answer on the free rung (45,000 seconds for “45000 ms”) failed the check and escalated, and the cheap rung resolved it correctly. Under READ authority, the resolved answer’s write was denied with the apply adapter called zero times; under WRITE, it applied once. ⁷

The scheduler matrix verifier asserts 16/16 code-versus-table parity at the v1 surface. The router track has a verifier waiting for a run that does not exist, which makes every router sentence above checkable in a simple way: each is design or status, never outcome. ^{5 13}

What this is not

• Not a ladder endorsement. The favored architecture failed its own adoption rule. Cheaper did not mean adopted.
• Not a top-first endorsement. Its clean wrong-acceptance column came, on the one decisive item, from a truncated call.
• Not a model ranking. Rungs are mechanisms with costs. The rule beat the strong model on one structured item because the operation was already specified.
• Not economics. Person costs are scenarios; model prices are one period’s declarations.

Where it is still weak

1. One run, one workload, one checker. Forty synthetic items with no repetition; nothing travels without a new run. ⁷
2. The adequacy gap is structural. A05 passes and T07/T10 fail by the checker’s design. ⁷
3. Decline-reason routing is untested. The zero paid-rung resolutions motivate it; nothing measures it.
4. Scheduler flags are precomputed and decisions unrecorded. The seam returns values; deriving and persisting them is the caller’s job. ⁴
5. No matrix covers epistemic-v2. The split-budget rows are source-inspected only. ¹⁶
6. The router track is unrun. ¹³
7. Pricing prefix shadow and sibling adapters. Named, not repaired. ¹⁷

Do this now

Thirty minutes. Put one model call on trial before spending it.

1. Write the next step of a real task as five flags: process budget, verification pending, proposals wanted, human authority needed, model budget. Run them through a function like the one above and write down the operation with its reason. Then change one flag and watch the priority reorder.
2. Take an escalation path you already run (retry with a bigger model, then ask a person) and record a decline reason on every transition for one day. Count how many escalations carried a reason no stronger model could fix.
3. Find one call your accounting could not see: an empty response, a timeout after billing, dropped usage. Decide what its row should have said, and whether your adapter parses usage before or after the step that can fail.
4. Write the falsifier for one architectural preference you hold. If you cannot name the thresholds that would make you revise it, you hold a slogan, not a bet.

If you are building with an assistant:

Decide the operation before the model: process stop first, pending
verification next, proposals as calls only while model budget remains, a
human gate before effects, otherwise stop. Keep model budget and process
budget as separate flags. Climb escalation ladders with a recorded decline
reason at every transition, and adopt a ladder policy only under a rule
frozen in advance that covers cost per accepted outcome, correct acceptances,
and wrong acceptances together. Parse usage before anything that can fail.
Present competing architectures with their falsifiers, and leave unrun
experiments unrun.

Failure modes

• Calling the next model by default. A declined rung is evidence about what to try, not an instruction to spend more.
• Reading one clause of the verdict. “Top-first won” and “the ladder was cheaper” are both true and both incomplete.
• Crediting the cutoff. A call truncated mid-deliberation delivered no judgment, whatever its reasoning was leaning toward.
• Tuning the checker after A05. Repairing adequacy after the fact destroys the evidence that measured it.
• Merging the budgets. One generic “budget exhausted” flag cannot express checks surviving spent cognition.
• Narrating the unrun. Design, implementation, and preregistration are not a result, however complete they look.

What this chapter established

• Operation, escalation, and selector are three decisions with three different evidence states: source-inspected policy, a measured stage, and an unrun design. ²
• The deterministic seam orders process stop, verification, budget-permitted calls, the human gate, and stop, with model and process budgets split, flags caller-supplied, and decisions unrecorded. ²
• The measured ladder was cheaper per accepted outcome ($1.061 against $1.453 at the $5 scenario) and produced one more correct acceptance, and it failed its frozen adoption rule because it added one accepted-but-wrong outcome. ⁷
• A05 shows a grounding checker accepting a grounded wrong answer; top-first avoided it through a truncated, unusable output, not demonstrated judgment. T07/T10 bound the checker from the other side. ⁷
• The failure-path usage defect is repaired in current source with regression tests; the bundle is untouched. The pricing-prefix shadow remains named. ⁹
• The model-router challenge is specified with a falsifier and remains unrun. ¹³

Next

The pieces are now all on the table, and this chapter deliberately does not assemble them: decision records, authority, observed effects, bound verification, replay, blind proposal collection, measured promotion, and an execution policy with recorded reasons. Each was tested against its own fixture, and several survived by being told no. Whether they hold together when one task has to pass through all of them — and which joints the runtime actually enforces — is the last construction question in the book. Chapter 30 then asks what the construction means once the process, and the policy deciding what happens next, is yours.

Continue with Applied AI.

References

• Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, Dor Muhlgay, Noam Rozen, Erez Schwartz, Gal Shachaf, Shai Shalev-Shwartz, Amnon Shashua, and Moshe Tenenholtz. MRKL Systems: A Modular, Neuro-Symbolic Architecture That Combines Large Language Models, External Knowledge Sources and Discrete Reasoning. arXiv:2205.00445, 2022. Paper.
• Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665, 2024. Paper.
• OpenAI. GPT-5 System Card. 2025. System card.

Implementation sources: Stage 29B ran at CodeAI 7a0d43b. Source inspection for this chapter used a1b562a, which contains the failure-path usage repair, its regression tests, the router track files, and a behavior-neutral reformat of scheduler.py. src/codeai/scheduler.py: Operation, SchedulerInput, SchedulerDecision, decide_next_step, POLICY_VERSION; src/codeai/runtime.py: Runtime.decide_next; src/codeai/ladder.py: run_ladder, _candidate, _check_decline, project_ladder_run, render_receipt; src/codeai/providers.py: OpenCodeCognitionAdapter.send, PRICING_TABLE, estimate_cost_usd; src/codeai/router_model.py, router_contract.py, router_experiment.py, router_analysis.py; experiments/router_compare.py, experiments/verify_router.py, experiments/router-prereg.md, experiments/router_cases_v1.DRAFT.json. Tests: tests/test_opencode_gateway.py (usage-preservation regressions), tests/test_budget_safety.py (pricing shadow). The scheduler excerpt is current source; the precedence fragments were executed as teaching code, not as a stage. The ladder traces are read from the preserved ledger events and analysis rows. Evidence: experiments/applied-ai/evidence/execution-ladder/2026-09-14-7a0d43b/ (preregistration, evidence report, analysis rows, preserved bodies and ledgers, independent verifier), experiments/applied-ai/evidence/scheduler/ (v1 matrix demo), and docs/applied-ai/ch28-router-experiment-design.md (unfrozen, unrun design). No evidence was modified. Footnotes mark provenance: measurement notes refer to pinned evidence, demo notes to preserved unpinned execution, source notes to inspected code, report notes to frozen designs and reports.