# The Complete Context System # Book: Context From First Principles # Chapter URL: https://programmer.ie/books/context/27-chapter/ # Prose fingerprint (sha256, normalized): 6561d6b03daaec4854eb416d5bf4e00c6619c68d1736a8e01a43f627585bae42 # Snapshot generated at build time by Hugo. This is the complete rendered prose # of the chapter; it is not a transcript of the source file. The book began with a developer’s eleven words and the organisation of information behind them, and asked how anyone would know whether changing the context had mattered. This chapter draws the system that came out of that question, and then asks of each part what it decides, what it only consumes, and what evidence stops where. “Complete” refers to the narrow end-to-end experimental path represented by the project’s components: supplied information, candidate construction, constrained assembly, host-runtime transport and behavioural evaluation. It does not mean that every link has been exercised together on the current versions. In particular, the candidate builder was added after the principal compiler and behavioural runs, the historical live qualification used an earlier adapter, and the Pi path has only a scripted-provider check. The system is complete as an architectural account of the investigated path, not as a concurrently qualified deployment or a complete solution to context engineering. flowchart TD A["Available information<br/>typed, supplied records"] --> B["Candidate builder<br/>observations, representations, projected verdicts"] B --> C["Context Compiler"] R["Request and policy"] --> C C --> D["Bundle + trace, or named failure"] D --> E["Render and transport envelope"] E --> F["Adapter injects one block<br/>into host model context"] F --> H["Observer: separate read-only path<br/>OBSERVED BOUNDARY"] H --> J["Reconciliation<br/>payload only"] D --> J F -.-> P["Provider request and processing<br/>not observed"] P -.-> G["Model: not observed"] G -.-> I["Behaviour"] I --> K["External evaluation"] The diagram combines implemented components, separately recorded experiments and an unobserved downstream path. Its solid arrows identify intended interfaces, not proof that one current-version run traversed them all. The observer records a host-runtime boundary; the reconciliation function checks payload properties but does not itself establish session pairing. The dashed path marks host-to-provider translation and model processing, neither of which the OpenCode instrument captures. Behavioural outcomes were measured in separate studies, not observed as part of one fully reconciled, current-version chain. The separations the book earned Each arrow above replaced a conflation that ordinary practice makes without noticing: information ≠ context available information ≠ observation observation ≠ candidate candidate ≠ admission eligibility judgement ≠ enforcement representation generation ≠ representation selection selected set ≠ ordered bundle bundle ≠ render render ≠ transport block transport block ≠ observed host context observed host context ≠ provider request provider request ≠ model use model use ≠ useful behaviour structural validity ≠ behavioural utility synthetic occurrence ≠ ecological prevalence Cost has its own chain: declared cost ≠ closure cost ≠ render estimate ≠ provider accounting. A retention class says what may be done to an item and a requirement says whether this computation must include it. The last few chapters added a second set, and it is the set most worth remembering: valid returned bundle ≠ complete feasibility search ≠ optimal bundle selection deterministic ≠ complete deterministic grader ≠ valid measurement separate implementation ≠ independent specification parity ≠ correctness reconciled ≠ session attributed influential ≠ helpful explicit ≠ correct From information to candidates The candidate builder is the newest stage, and it deserves the same scepticism as the rest. The builder does not discover or verify source information. It maps typed records supplied by a caller—files, instructions, tool calls, working notes, memory items and retrieved passages—into observations carrying source identity, locator, version, payload and provenance, rejecting malformed records individually. In the worked fixture, eight supplied records produce nine observations because one tool call is represented by both a capability definition and an execution result; these yield twenty-nine candidate forms. Those counts describe this input and the builder’s expansion rules, not the quantity of independent evidence or the completeness of retrieval. The trace’s “sources consulted” categories name supplied source kinds; they do not establish that any repository, memory store or external system was searched exhaustively. Each observation can yield several representation candidates: full supplied text; a compact version produced by deterministic word truncation and explicitly marked lossy; an anchor made from a locator, version and first line; and a pointer-like reference. The compact form is not a semantic summary, the anchor is not the complete observation, and the pointer is not the referent. No model is invoked to construct these forms. More seriously, the reference path has an unresolved operational contract: its text names an anchor for recovery, but the compiler treats forms sharing one content identity as mutually exclusive, so a same-content anchor cannot be admitted alongside that pointer under the current rule. Neither resolvability nor preservation of the evidence needed to act on the pointer has been demonstrated end to end. The trace can record intended lineage without making the reference usable. pointer present ≠ referent content present Identity is layered, but the identities answer different questions. A candidate identifier is derived from source identity, representation form and content, which makes the tested build stable under a change in supplied-record order. Forms of one observation share a content identity, while source and observation identities remain distinct. A stable hash shows equality under the chosen identity recipe; it does not prove source equivalence, freshness, provenance authenticity or that a later renderer and transport envelope carry identical model-visible text. Reproducibility also depends on preserving the recipe and implementation version. Judgements are projections The builder fills the compiler’s eligibility fields, and its provenance labels say where each came from: Field Source of the value Unknown handling Scope Declared scope compared with the request’s active scope (simple equality) Unknown fails closed Freshness A supplied validator status or supersession flag; timestamps are carried and never compared Unknown is eligible under the default policy, with the reason recorded Authority A policy table over source class Unknown class fails closed Requirement Declared by the source, else a policy default by source type Unknown becomes discretionary Floor Policy, from the source’s retention Unknown retention gives floor zero, visible as unknown Dependencies An explicit edge from a tool output to its tool definition A missing definition leaves the output standing alone, recorded as unknown Cost A word-based estimate Unknown cost is not representable Version A revision for files; a constant for other source types Missing file revision is rejected Unknown is not handled by one system-wide principle. The current builder rejects unknown scope and authority classes, accepts an unknown freshness status under its default policy with a recorded qualification, permits an unknown retention class through a zero floor, and leaves an output without a known tool definition standing without an enforceable dependency. A word-based cost estimate is always produced, so unknown cost cannot be represented as a separate state. These are different policy choices at different boundaries, not evidence that the unknown values are safe. Trace visibility can help an auditor notice a permissive decision but does not repair a missing dependency or validate freshness. These are narrow deterministic projections of the richer judgements of Chapters 19 to 21, sufficient to populate the compiler’s schema for the source classes tested. A source-class policy establishes a configured permission and not real-world authority. Scope by equality does not solve branches, worktrees or multi-world tasks. Freshness from a supplied status does not solve semantic freshness. The projection is where an epistemic gap sits, and the deterministic enforcement downstream does not close it: explicit ≠ correct authority policy ≠ authority truth Relationships, and one limitation that is the builder’s own Dependencies, conflicts, alternative forms and coupled groups are distinct relations, but the implementation represents them unevenly. A tool output may depend on its exposed tool definition. Two versions of a locator with different payloads can trigger a structurally identified conflict group with a marker; the group is required only when all its members pass the gates. In the stale-versus-current fixture, the marker is itself rejected and the current revision travels alone, leaving the recorded disagreement in the trace rather than preserving it as an admitted conflict group. That may be correct for a task requiring only current state, but it is not a general resolution of historical disagreement. The detection key is assigned by source-specific adapters and does not establish that two natural-language claims contradict each other. Conflict detection, eligibility and preservation remain separate obligations. The important limitation concerns representation. Several forms of one content are emitted, and the compiler admits at most one, because it infers mutual exclusion from a shared content identity. Chapters 18 and 22 showed that same content identity ≠ automatic mutual exclusion: forms may be substitutes or complements, such as a summary with an exact fragment. The one-form rule is an implementation limitation and not a law, and complementary forms are unsupported until the schema has an explicit relation for them. Numeric representation ranks are likewise policy order and not semantic quality. What the builder demonstrates Within its tested typed-input domain, repeated builds and input-order reversals produced the same candidates, and automated tests exercise the builder’s identity, provenance, representation lineage, scope, freshness, authority, conflict, dependency and import boundaries. In one constructed case its output composes with the compiler and yields a bundle accepted by two structural validators. That establishes a bounded integration path for those fixtures, not independent validation of the validators’ shared assumptions or of the builder’s world-facing judgements. It does not measure relevance, semantic preservation of truncated content, live-source correctness, actual temporal validity, delegated authority, usable reference recovery or behavioural utility. The worked-case counts are demonstrations, not effect estimates. From candidates to a bundle The compiler decides membership, representation and order for one computation under explicit constraints, and it returns either a bundle with a trace or a named failure. It consumes the eligibility verdicts, requirements, floors, dependencies and groups the builder attached, and it enforces them. The evidence has three separate strands: soundness: a returned bundle satisfies the enforced constraints completeness: if a legal bundle exists, the compiler finds one optimality: among legal bundles, it chooses the best The compiler matched the expected success-or-refusal status for forty-two historical synthetic compilations: thirty-four successes and eight refusals. A later finite enumeration found legal bundles for those thirty-four and no legal bundle for the eight, under an audit that shared the compiler’s token-cost and one-form-per-content assumptions. That is useful finite evidence, not a general completeness theorem or a validation of the richer complementary-representation model. A separate constructed counterexample exhibits a false refusal: the greedy procedure cannot fit its chosen path although another permitted combination fits. That directly defeats completeness for the tested constraint model. No optimality is claimed; a 670-token reference was chosen over an admissible 400-token full form in one case. The separate checker validates returned bundles, but it shares the compiler’s data types and its one-form assumption, its cost model is weaker than the engine’s, and it checks consistency and not the correctness of reasons. It cannot prove feasibility. A second implementation in another language reproduces the first on the frozen corpus, and parity reproduced three defects with the reference. Reproducibility can faithfully reproduce a mistake. A trace records the decisions the implementation says it made, which makes inspection possible and does not make the decisions right. From bundle to observed runtime context A render is the bundle’s text; a transport block adds an envelope; and the adapter attempts to insert that block into a host-runtime context. A separate read-only observer code path, running in the same process and plugin package as the injector, records the host context. The pure reconciler checks exact text relationships: the block appears once in the observed system material, the recovered render matches the emitted render, the marker count is correct, and the requested model matches the host-observed model when that identity is present. It does not independently attest session pairing, provider receipt or the model actually using the material. Thus the strongest claim is an exact-text observation at a specified host boundary, conditional on the harness’s separate attribution checks—not end-to-end delivery to the model. That is a narrower claim than it sounds. The observer is a separate code path and not a separate failure domain. Reconciliation does not check session identity, which the surrounding harness pairs. A single live compile-to-observe qualification is recorded for an earlier compiler and adapter combination, so it is historical evidence for that configuration and not a qualification of the current one. The provider request was not captured, and nothing observes the model. A live no-context negative control was recorded in the second live wave for one configuration, and it has not been repeated on the current one. The newer Pi adapter preserves the conceptual split between deterministic bundle construction, prompt-section insertion and observation, but it changes the host boundary and the available evidence. Its check uses a real Pi SDK session with a scripted provider, allowing a comparison between the observed effective prompt and what that scripted provider received, including a negative test for a later whole-prompt override. That is stronger than inspecting an earlier hook alone for this scripted path, but it is not independent receipt by a production provider and is not evidence of real-model use. The historical OpenCode qualification does not transfer automatically across host implementations or code revisions. From treatment to behaviour Two behavioural experiments exist, and they answer different questions. The first supplied frozen bundles to a local reader through a chat interface, so the live transport path was not involved. On six constructed tasks, moving from no context to the compiler’s bundle changed every full parsed response and four of six chosen actions, improved two scores, worsened one and left three unchanged. Wording varied even between repeats on an identical bundle, so the decision-level results are the more informative. Compared with hard-gated greedy, no decision or score changed, but three of six pairs received digest-identical bundles and the other three sat at ceiling, at the floor or were solved by every supplied bundle, so the experiment had limited power to discriminate. Compared with the hidden-evidence oracle, no decision or score changed, and four of six pairs were identical. That oracle is a fixture oracle and not a behavioural optimum. One task showed a harmful bundle-level association: no context held, and every supplied bundle and surgical variant released. The parent already sat at the behavioural floor, so candidate-level cause is unresolved. The calibration task was not a clean negative control, because the prompt and the grader disagreed about the response field and every condition scored zero, including no context. One same-bundle observation shows two readers acting differently on an identical bundle, and it is exploratory. A deterministic grader repeated every recorded score, and it is still not a valid measurement of a task that named no field. The live-runtime study consists of two waves with different validity. The first executed twenty-four runs but yielded no transport-valid observations because the instrument produced no usable records; its identical scores cannot answer the intended context question. The repeated wave used eight fixtures crossed with no context, a matched distractor and exact minimal evidence. All twenty-four conditions passed the host-boundary manipulation checks, including observed absence of the experimental block in no-context runs; the reader was pinned by digest and two preflight canaries were required. The supplied evidence was authored from fixture truth rather than chosen by the compiler, so this evaluates a delivery-and-reader intervention, not the compiler’s selection policy. Every condition scored zero, with the same two fixtures flagged harmful and no measured evidence use. The recorded description calls this a behavioural failure under verified delivery, with “verified” confined to the observed host boundary. verified delivery ≠ measured treatment effect The repeated wave found no difference on the recorded scores, harm flags or evidence-use measure for the reader, eight fixtures and observed host configuration. It does not establish that responses, actions or internal processing were identical: the raw responses were not preserved in the frozen result and action-level equality was not checked. Because all scores were at zero, that instrument could not detect a further decline and was poorly placed to distinguish helpful from harmful context. Decoding used reader defaults, an unrelated plugin failed to load, and the installed compiler package pin changed during the wave even though the injected fixture blocks did not depend on compiler selection. These qualifications bound the result; they do not convert a measured null into evidence of no possible context effect. Neither experiment supports the claim that the compiler improves behaviour, and neither estimates prevalence. Four kinds of evidence, and what each has Kind Question What exists Boundary Candidate construction Can typed inputs be turned into inspectable candidates? Builder tests, worked case, determinism checks Supplied typed records; projected verdicts Structural Is a returned bundle legal under the stated rules? 42-condition corpus, validator, parity; an exploratory finite feasibility audit and constructed counterexample, not yet frozen in the evidence register Synthetic; supplied eligibility; known completeness and representation gaps Transport Did the identified treatment reach the observed boundary? One historical live qualification, earlier probes, a verified live wave One host boundary; a same-process observer; current configuration not requalified Behavioural Did the change alter decisions or outcomes? A six-task direct-reader ladder with surgical, repeat and transfer probes; a 24-run live wave Constructed tasks; one primary reader; scores at floor in the live wave Ecological How often do these situations occur in real work? None The corpus of real sessions is empty The system described here is a version-spanning synthesis rather than one frozen build exercised from source record to measured outcome. The construction corpus and direct-reader ladder, early runtime probes and live qualification, trace repairs, later adapter changes, candidate builder, and subsequent finite feasibility audit and rescoring belong to different recorded stages. Their results should be attributed to the exact code, fixture, model, policy and instrument versions used, wherever those identities are preserved. A later implementation may satisfy the same interface and still require renewed qualification. In particular, successful builder-to-compiler composition and historical host transport must not be read as one demonstrated current-version builder-to-model experiment. current implementation ≠ historically measured implementation Policy is not law Mechanism Structural requirement Current policy choice Evidence Eligibility Hard, non-negotiable gates Booleans projected from source class, status and equality Fixtures; no real traces Obligation Hard against soft Four requirement classes Two obligation classes fail identically Selection among options Explicit objective within the legal region Relevance, then coverage, cost, identifier A baseline only Stopping May stop below the ceiling Relevance threshold of 0.3 A policy value Representation Substitute against complement One form per content identity A known limitation Floors Preservation obligations Numeric rank comparison A convention Ordering Membership separate from order Fixed role order Policy default Render repair Never break hard constraints Drop the last discretionary item Estimated units Feasibility Report failure honestly Greedy admission Counterexample exists Legal policy is not good policy, good policy is not optimal, and neither generalises. The behavioural nulls fit that reading, and they do not show the policies are wrong. The book also examined pruning, compaction, fading, externalisation, recall, cache-aware rewriting and coverage-aware allocation. Those chapters supplied mechanisms, design hypotheses and selected external evidence, but this project’s end-to-end behavioural experiments did not establish their comparative value. It would be too strong to say that every mechanism is unsupported in the literature, and equally too strong to treat a plausible mechanism or a small synthetic demonstration as a justified production subsystem. For each one, the remaining question is whether real workloads exhibit the relevant failure, whether the mechanism changes it under controlled conditions, and whether the resulting benefit exceeds its added cost and failure surface. The claim ledger Claim Evidence Status The builder constructs deterministic candidates for its tested source classes Automated tests, determinism checks Established within tested boundaries The builder’s scope, freshness and authority judgements are right None Not established Returned bundles satisfy the enforced constraints Conformance, validator Established on a finite corpus, with shared-assumption limits The compiler is complete A counterexample Not established The compiler is optimal None Not established Complementary representations are supported None Unsupported Transport works One historical qualification, a verified live wave One configuration; host boundary only Provider-wire or model receipt None Not established Context can alter decisions Six-task ladder Established as cases Verified delivery of oracle context produces a measured effect Second live wave, all scores at zero Not established; a null on the measured variables The compiler improves behaviour None Not established Readers differ on identical bundles One observation Exploratory These mechanisms are common in practice None Not established What a reader can build The approach does not require adopting this compiler. Start by observing, without mutation, what a particular host exposes at a clearly named boundary. Make the provenance and derivation of candidate judgements inspectable; preserve unknown rather than silently treating it as true. Give the assembler explicit constraints, representation obligations, dependency information and a declared budget, then require either a checkable bundle or an explicit account of why its procedure could not produce one. Keep the decision trace separate from independent structural checking. In behavioural tests, pin the reader and task, verify the assigned context condition actually reached the observed boundary on each run, and compare outcomes using graders validated for the task. An assembler’s refusal is not a proof of infeasibility unless feasibility has been established independently. These practices form a testable method, not a requirement to use one framework. The complete viewer The Context Viewer now shows the whole chain in one place: available information to behavioural evaluation, with compiler decisions and viewer diagnostics side by side and every evidence stop marked. Open the lifecycle scene and move backwards along lineage from any admitted item — why it entered, what changed its representation, where it was observed, and where measurement stops. The complete chain with recorded, referenced and open stages distinguished. Displaying separately collected records together is not an end-to-end qualification. Open the complete Context Viewer The Context Compiler answers: “What should be assembled for this computation, under this policy?” The Context Viewer answers: “What was available, what was selected, what changed, what was observed, what might be wrong, and what evidence supports that conclusion?” Where the argument ends Context engineering is the construction, delivery and evaluation of a bounded representation of available information for a particular computation. Construction quality is not delivery, and delivery is not utility, and the project has made those separately measurable. It did not find the right context. It made the stages of context construction explicit, inspectable and separately falsifiable. The next work should follow the demonstrated gaps rather than the attraction of more machinery. First, pin the finite feasibility checker and false-refusal counterexample, then make substitute-versus-complement relations explicit and test the reference-recovery dependency that currently fails to close. Independently validate builder judgements, including unknown scope, freshness, authority and missing dependencies. Requalify the present runtime adapter with a live positive canary, an observed no-context negative control and enforced session attribution. Only then run discriminating behavioural fixtures with pinned reader identity, stable decoding, preserved raw responses, valid calibration tasks and bundles differing on the mechanism under test; include exact same-bundle comparisons across readers and candidate-level removal/restoration outside floor and ceiling cases. Finally, collect bounded, privacy-reviewed real-session traces to estimate how often the proposed mechanisms are needed. Record code and policy versions, source and candidate lineage, selected and rejected material, rendered and observed identities, transport receipts, outcomes and exclusions. That record should permit an auditor to say not merely that something changed, but which boundary was observed and which claim remains unearned. References Project Context Pi. First-party local source package, with pinned SDK declarations and implementation provenance. The structured prompt-section path is checked in a real SDK session with a scripted provider; historical OpenCode results are not extended to the new host. Project Context evidence register (companion repository, evidence/README.md and its manifests). Consumed here by short name: Compiler construction, Implementation parity, Live compile-to-observe qualification, Earlier runtime qualifications, Instrumentation failure, Matched behaviour, Reader-transfer probe and the requalified oracle-leverage wave. Identifiers, code revisions and digests are held there and not in the text. Project Context Compiler, Project Context OpenCode and Project Context. Source repositories, Apache-2.0, first-party code by this book’s author: the candidate builder and compiler, the runtime adapter with its observer and reconciliation, and the research and evaluation repository. https://github.com/ernanhughes/project-context-compiler, https://github.com/ernanhughes/project-context-opencode, https://github.com/ernanhughes/project-context