Part 2 — Get the Model Out of the Chat Box

Tuesday, three times

Monday evening, on your laptop, you work through a design problem with a model. Forty minutes of back-and-forth. You settle three decisions and rule out two approaches for reasons that took a while to articulate.

Tuesday morning, on the train, you open the same app on your phone. It is a new conversation. You paste in what you can remember.

Tuesday afternoon, in your editor, a different assistant offers to implement something you explicitly ruled out last night, confidently, because it has never heard of last night.

Three surfaces with separate contexts and no shared memory. And the only thing carrying continuity between them is you, retyping.

This is Chapter 1’s problem again, one level up. There, the human was the integration layer between a chat window and an application. Here the human is the integration layer between their own tools. We removed the person from inside a single exchange and left them holding everything between exchanges.

What would a system look like that was actually built for the way a person works?

The diagnosis

The reason your context dies at the surface boundary is not a missing feature. It is where the state lives.

Every chat app keeps its own history. Every IDE plugin keeps its own session. Every browser extension keeps its own tab. Each one is a little process with its own memory, and switching between them is a process restart with no serialization.

If your state lives in the interface, you have as many processes as you have interfaces.

That sentence is the whole architectural argument of this chapter, and everything else follows from inverting it. The surface should be a view onto a process, not the process itself. One runtime; many windows.

The consequences of the inversion are immediate and concrete:

• The durable record is the source of truth, not the UI. A surface can close or drop mid-task without losing work.
• Identity and authority travel with the request, rather than being assumed by whichever app is open.
• Adding a new surface — a phone, a terminal, a voice interface — is adding a client, not building a second system with a second memory.

This is the same move the book has been making since Chapter 1, applied one level out: separate the thing that thinks from the thing that owns the work, then separate the thing that owns the work from the thing you happen to be looking at.

Five properties, and where each one comes from

Here is the system, stated as requirements. What matters is not the list — anyone can write a list — but that every item is forced by an argument already made. A design that cannot say why each property is there is a wish list.

The rest of this chapter takes them one at a time, and then names the places where they fight each other.

Works wherever the person is

The target here has been articulated before, and well. Kleppmann, Wiggins, van Hardenberg, and McGranaghan set out seven ideals for local-first software: fast with no network round-trip to do work, multi-device, offline-capable, collaborative, long-lived so your data survives the vendor, private, and user-controlled (Kleppmann et al., 2019).

Read that list against the AI tools you currently use. Most fail on longevity, privacy, and user control immediately; several fail on multi-device; nearly all fail on offline.

Those seven ideals are the direction of travel for what we build, and it is worth being honest about the distance. Full local-first — conflict-free replicated data types, seamless offline collaboration between multiple people — is a serious distributed-systems undertaking and this book does not pretend to deliver it. What we build is the tractable core: a single durable store that every surface reads and writes, with the ledger as the authority. Multi-device and longevity come almost free from that. Offline and collaboration are real work we will name rather than hand-wave.

Never loses context

This property is the one most commonly misread, and getting it wrong is expensive in both directions.

“Never loses context” does not mean “put everything in the prompt.” Two independent reasons say so.

The first is measured. Liu and colleagues examined how models actually use long inputs on multi-document question answering and key-value retrieval, and found a U-shaped performance curve: models use information best at the beginning and end of their context and noticeably worse in the middle. Smaller models showed mostly recency bias; larger ones showed both primacy and recency. Their conclusion is the one that matters here — increasing the context window does not guarantee robust access to what is in it (Liu et al., 2024).

The second is Chapter 6’s. Context costs money per token, on every call, forever.

So the design separates two things that get conflated constantly:

Memory is durable. Context is selected.

The ledger remembers everything. The context compiler decides what a specific call is allowed to see, in what order, within what budget — and then records that decision and hashes the resulting package.

That last part is what turns a convenience into an engineering primitive. If the package is hashed, the call is reproducible; if the selection is recorded, you can ask later why the model did not know something. Which yields a more precise version of the property:

You never lose context. You deliberately exclude it, and the exclusion is written down.

The difference between deliberate exclusion and accidental loss is the difference between a system you can debug and one you can only apologize for.

A small executed selection shows what the record looks like. Compiling five items against a 1,500-token budget — the agreed objective declared required (300 tokens), two recent decisions (400 each), a week-old design note (900), and a long thread dump (2,400) — returns a package holding the objective plus the two decisions, with a trace like:

package 05ec9a95… (budget 1500)
  included: objective — included because required
  included: decision-1, decision-2 — included within budget
  excluded: design-note, thread-dump — excluded because budget

(Fragment executed offline against the current ContextCompiler; identifiers shortened.) The package carries its own identity (ContextPackage.package_id) and the trace records included_ids, excluded_ids, and a per-item reason (CompilationTrace.entries). Reopen the package next week and you can answer the question the Tuesday story could not: the model did not know the design note because the compiler left it out, within which budget, for a stated reason — not because a surface forgot it.

Always remembers history

“Remembers history” needs decomposition or it becomes a database with no schema. The cognitive-architecture literature offers a useful one: Sumers, Yao, Narasimhan, and Griffiths describe language agents in terms of a short-term working memory plus long-term episodic (experience), semantic (knowledge), and procedural (how things are done) memories, with a decision procedure that cycles between planning and execution (Sumers et al., 2024).

Mapped onto what we build:

Two reasons this is not optional, both already argued. Chapter 5: when a failure is finally discovered, a system with no record has an unbounded blast radius, and one with a record has a bounded one. Chapter 8: you cannot measure arrival — started versus finished, rework rate, backlog age — without a history to measure it over.

And a third that only appears once the system exists: a critic you cannot re-score is a critic you cannot change. Chapter 7 required the critic be pinned and versioned, with the archive re-scored when it changes. That is impossible without preserved artifacts. The memory is what makes the measurement survive its own instruments.

Best, cheapest, most available

Those three words pull in different directions.

• Best is capability: can this model do this task at an acceptable rate?
• Cheapest is price: what is the least I can pay for that?
• Most available is reachability: is it up, am I rate-limited, what is the latency from here, is this region allowed?

That third constraint gets left out of most discussions and it is the one that wakes you at 3am. A router optimizing only capability and price will route everything to one provider and fail completely when that provider has an incident. Availability is not a tiebreaker; it is a first-class input.

So the router decides over all three, using the ratchet from Chapter 6 — floor first, escalate on failure, route by expected difficulty, distill what you can own — and subject to Chapter 1’s second bet:

The router’s own decision path contains no model call.

A stochastic component deciding how to spend money on stochastic components is not a cost control; it is a second thing to debug. The decision is a deterministic function of recorded state: what the task requires, what the budget allows, what the policy permits, what is currently reachable. The ambition, flagged now so that it can be checked later, is that the router be a specification the runtime interprets rather than a program. Chapter 28 reports what survived contact with implementation — and what did not.

Reviews every contribution

Chapter 5 established that review cannot be a matter of resolve, because complacency defeats experts and does not yield to practice. Chapter 7 established what a critic must satisfy to produce a number worth having. This property is those two chapters made structural.

Three layers, cheapest first, in the order Chapter 7’s ladder demands:

1. Mechanical checks on everything, automatically. Does it parse and compile; do cited sources resolve and quoted spans exist verbatim; do the numbers sum. Free, exact, and applied to every contribution without exception.
2. An independent critic panel on everything. Different model families from the generator, blinded to provenance, pinned and versioned, with its measured agreement against a gold set attached to every score it emits. Hold “independent” loosely until it is measured: different families can still make the same mistake, and Part 5’s first experiment finds exactly that.
3. Human attention, routed by risk. Not spread evenly — concentrated on what is irreversible, novel, or high-consequence. Chapter 20’s authority machinery is as much an attention router as a permission system.

And the part that closes the loop: the system measures its own review. Seeded defects, periodically, with the catch rate recorded over time. A review process that has never been tested is a review process with an unknown value.

The shape of it

    flowchart TB
    subgraph S["surfaces — views, not the system"]
        S1["terminal"]; S2["editor"]; S3["browser"]; S4["phone"]
    end
    subgraph R["runtime — one process, deterministic"]
        CC["context compiler<br/><i>selects · orders · hashes</i>"]
        RT["router<br/><i>capability · price · availability</i>"]
        PO["policy<br/><i>authority · budget</i>"]
    end
    subgraph W["workers"]
        M["model call<br/><i>the only stochastic step</i>"]
        T["tools / actions"]
        V["verifiers<br/><i>mechanical + critic panel</i>"]
    end
    subgraph L["durable memory — the source of truth"]
        LE["ledger<br/><i>episodic</i>"]
        AR["artifacts<br/><i>raw bytes, hashed</i>"]
        CL["claims<br/><i>semantic</i>"]
        SP["specification<br/><i>procedural</i>"]
    end
    S --> R
    PO --> RT --> CC --> M --> V
    M --> T --> V
    V --> L
    L --> CC
    SP --> PO
    L -.->|"routed by risk"| S
  

Read it once and notice what is not in the middle. There is no conversation. The transcript is not a component. What flows between the parts are compiled packages and recorded results, with bytes preserved and evidence attached.

Where these properties fight

A design chapter that lists only benefits is marketing. These five requirements conflict, and the conflicts have to be decided rather than discovered.

Cheapest versus constantly reviewed. Review costs tokens from the same budget as the work. A critic panel on every contribution can plausibly cost more than the contribution did. This must be an explicit split — what fraction of spend is evaluation? — decided up front, not an emergent property of whatever anyone added last.

Works everywhere versus remembers everything. One durable store reachable from four surfaces is a synchronization problem, and synchronization is where distributed systems go to be difficult. Every honest local-first ideal here costs engineering to honor.

Never loses context versus the context budget. You cannot carry an unbounded history into a bounded, priced, U-shaped context window. Selection is mandatory. The design commitment is not “keep everything in view” — it is “record every exclusion.”

Remembers everything versus privacy. This one deserves more than a bullet. A system that records everything you do, across every surface you use, is a revealing artifact in its own right — more so than any single tool it replaces. Where does it live and who can read it? What is the retention policy for breach, subpoena, departure, or acquisition? Local-first’s privacy and user-control ideals are the right frame, and this must be a design input from the first commit rather than a compliance conversation in year two.

One runtime versus one point of failure. Consolidating state is what makes the rest work, and it also means there is now exactly one thing whose loss is catastrophic. Backup and a documented recovery path are part of the design, not operations trivia.

What gets built, and where

The reference implementation is CodeAI. At the point this chapter was written it was a subset of this design: a ledger, a context compiler with isolation seals, claims with evidence classes, policy and authority, and experiment machinery, but no surfaces and not yet the router described here. The book builds much of the gap, and says so when it is building rather than describing. Not all of it: the surfaces are never built here. They are left to later labs — and, as Chapter 30 argues, to you: the surfaces most worth building are the ones shaped around your own work.

The field has converged on the same layer — under a different name

In 2026 “harness engineering” independently became the industry’s name for much of this design space. Böckeler describes a harness as everything in an agent except the model itself, and sorts its controls two ways: guides, which steer the agent before it acts, versus sensors, which observe afterward so it can correct itself; and computational controls (deterministic and fast: tests, linters, type checkers) versus inferential ones (model-based review and judging, slower, costlier and less deterministic) (Böckeler, 2026). A source-code study of eleven coding agents — among them Claude Code, Codex CLI, Gemini CLI, OpenCode, Aider and OpenHands, about four million lines in all — describes seven canonical subsystems of such harnesses and reports that none of the agent runtimes imports a general-purpose agent framework and none retrieves code with vector embeddings: the field runs on hand-written loops and deterministic retrieval (Barbaste et al., 2026).

That is independent convergence on this chapter’s object, and the book keeps its own vocabulary deliberately. “Harness” names the layer around the model. This book is concerned with the process inside that layer: durable state with identity, authority separate from capability, evidence with provenance, verification bound to the state being accepted, safe repetition, and deciding what happens next. Guides and sensors map naturally onto context assembly and checks; the parts they do not name — the record of what happened and the decision that work is complete — are where Chapters 5 and 14 locate the hardest problems. Treat harness engineering as the external term for the same design space, and this book as one attempt to make the process inside it explicit and checkable.

Do this now

Twenty minutes, one diagram. Find where your state lives.

1. List every surface you currently use to work with AI — chat app, editor plugin, terminal, phone, browser.
2. For each, write what it remembers and for how long. Be exact: does it survive a restart? A week? A device change?
3. Draw the arrows that represent you carrying context between them. Count them.
4. Now mark, for the last thing you worked on: where is the record of what was decided, and could another process read it?

If the answer to step 4 is “in the scrollback of one application” or “in my head”, you have located the problem this part of the book exists to fix. Keep the diagram. By the end of the book you should be able to redraw it with every arrow that was you replaced by a record another process can read.

Failure modes

• Building the state into the interface. Guarantees one process per surface and a human carrying context between them.
• Confusing memory with context. Durable memory and per-call context are different mechanisms with different costs; conflating them produces either amnesia or enormous bills.
• Assuming a bigger context window fixes recall. Performance is U-shaped; the middle is where things go to be ignored.
• Losing context silently. Exclusion is fine. Unrecorded exclusion is a bug you cannot reproduce.
• Routing on price and capability alone. Availability is a first-class constraint, and you discover this during an incident.
• A router that calls a model. Adds cost and a second stochastic failure mode to the thing meant to control both.
• Reviewing everything equally. Attention is scarce; spend it where action is irreversible and novel.
• Treating the recorded history as an engineering detail. It is a privacy artifact from the first write.

What this chapter established

• The reason context dies between tools is that state lives in the interface. One runtime, many windows inverts that: surfaces are views, the durable record is the source of truth, and authority travels with the request.
• Five properties, each forced by a prior argument rather than asserted: surface independence, selected context, durable history, a deterministic router, and constant review.
• Local-first’s seven ideals — fast, multi-device, offline, collaborative, long-lived, private, user-controlled — are the direction of travel; the tractable core is one durable store every surface shares.
• Memory is durable; context is selected. You never lose context — you deliberately exclude it and record the exclusion. Bigger windows do not fix this, because recall is U-shaped.
• History decomposes into working, episodic, semantic, and procedural memory, and maps onto context packages, the ledger and artifacts, claims, and the specification.
• “Best, cheapest, most available” is three constraints, and availability is the one that causes incidents. The router decides over all three and contains no model call.
• Review is three layers — mechanical, critic panel, risk-routed human — plus a seeded-defect measurement of the review itself.
• The properties conflict: review competes with work for budget, sync is genuinely hard, context must be excluded, and a system that remembers everything is a privacy artifact from day one.

Next

This design has a dimension in space — many windows onto one runtime. It also has a dimension in time, and that one is easier to miss.

The component in the one stochastic slot will not stay put. Later models will be substantially different machines, the “same” named model drifts between snapshots, and an upgrade that raises the average can break the specific thing you depended on. Most software is built on a foundation that holds still. The next chapter is about building on one that doesn’t — and about why the frame described here is what makes that survivable.

Continue with A Revolver, Not a Foundation.

References

• Martin Kleppmann, Adam Wiggins, Peter van Hardenberg, and Mark McGranaghan. Local-First Software: You Own Your Data, in spite of the Cloud. Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! ‘19), pp. 154–178. https://doi.org/10.1145/3359591.3359737
• Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, vol. 12 (2024), pp. 157–173. https://doi.org/10.1162/tacl_a_00638
• Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive Architectures for Language Agents. Transactions on Machine Learning Research, 2024. https://arxiv.org/abs/2309.02427
• Birgitta Böckeler. Harness Engineering for Coding Agent Users. martinfowler.com, 2 April 2026. https://martinfowler.com/articles/harness-engineering.html
• Paul Barbaste, Tristan Darrigol, Germain Vu, and Tom Wiltberger. Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents — A Source-Code Study of Eleven Systems. arXiv:2609.00006 (v1), July 2026. https://arxiv.org/abs/2609.00006