Memory
Chapter 02 of 17

The Measurement Instrument

By AI, Software Engineering

Concepts

Chapter 2 — The Measurement Instrument

Source: 02-chapter.md

Organising question: what would an instrument have to observe in order to tell whether an AI memory system is actually good?

What this chapter is really about

Underneath the benchmark design, this chapter is about what makes an empirical claim about memory trustworthy — and, more sharply, about the adversarial relationship between a benchmark and the systems it measures. Every control (hidden labels, frozen runs, leakage audits, contamination dating) exists because a sufficiently motivated system-builder — or a sufficiently capable optimiser — will climb whatever gradient the instrument provides, including gradients that point away from memory. The chapter is therefore doing measurement theory for a domain where the measured object (behaviour change from history) is easy to fake and hard to isolate. Its deepest question: can we construct an instrument whose scores cannot be improved without improving memory?

Current thesis

Explicit claims

  • Demos fail as evidence in four specific ways: case selection, single phrasing, fluency judging, no reruns.
  • Two corpora with different jobs: controlled (mechanical ground truth) and real (ecological validity); scores never merged.
  • Each question family needs its own answer shape and metrics.
  • Baseline ladder plus ablation for every experiment; frozen run contract for comparability.
  • Failure attribution across nine categories including evaluator/label defect.
  • Six named leakage controls plus contamination dating for published query sets.
  • Two deliverables: the Memory Measurement Instrument and the memory system, developed together; Chapter 2 establishes instrument v0.1 and later chapters extend it.
  • A v0.1 measurement model with named dimensions (preservation through cost) reported as a scorecard, never a single number.
  • Mechanical scores and model-judged scores kept strictly separate, with grader identity in the run manifest.
  • A runnable scorer package with deterministic scorers, frozen-run manifests, controls, and a canned end-to-end demonstration (implemented, experiment pending).

Implied claims

  • Pre-registered Type-A/Type-B outcome pairs are sufficient to keep the author honest — the experiment slots function as intellectual commitment devices.
  • The ledger’s ontology (event, decision, validity interval, supersession, task, status) is adequate to generate all needed scenarios; no scenario the book needs falls outside it.
  • Human adjudication of the real corpus, with disagreements preserved, is trustworthy enough to anchor ecological validity.

Not yet established

  • That the listed controls are complete — leakage channels the author hasn’t imagined are, by definition, unaudited.
  • That per-question metrics actually measure distinct capabilities rather than differentially difficult views of one capability (the Q1/Q2 divergence is predicted, not shown).
  • That a synthetic generator can avoid baking in exploitable regularities beyond the ones the real corpus catches. The real corpus catches some generator artefacts; completeness is assumed.

What the chapter already gives us

  • The four-defect demo critique. Portable and devastating; each defect maps to one instrument feature. This is the chapter’s best teaching asset.
  • The two-corpora doctrine. Mechanical truth plus ecological resistance is a genuinely careful design, and the no-merge rule prevents the standard sin of averaging away the distinction.
  • The leakage catalogue. Answer, timestamp, template, lexical, query-duplication, evaluator-privilege, memorised-pattern leakage — each is a concrete generator requirement, not advice. This is rare: most benchmarks discuss leakage abstractly.
  • Evaluator/label defect as a failure category. Treating harness error as a first-class outcome (fixed by benchmark versioning, not silent patching) is the honesty mechanism most benchmarks lack.
  • Type-A/Type-B pre-registration. Stated in advance per experiment, including the revision each outcome demands. This is what makes the pending slots scientifically live rather than decorative.

Where the current treatment stops

  • The chapter assumes the ledger ontology is sufficient but never argues it: are there memory phenomena (entity identity over renames? graded commitment? multi-agent disagreement?) the ledger cannot plant? The generator’s expressiveness bounds the book’s falsifiability.
  • Real-corpus adjudication is hand-waved: who adjudicates, with what agreement rates, and what happens when adjudicators disagree about validity intervals (the hardest labels)? “Disagreement preserved” is a slogan until there is a protocol.
  • Contamination handling is noted but thin: reporting query-set publication dates relative to training cutoffs does not solve contamination for models trained on the open web after publication; held-out variants are promised but undesigned.
  • The frozen-run contract freezes prompts, models, budgets — but the evaluator (human rubrics, model judges) also drifts. Model-graded evaluation with versioned judges is never discussed, though Chapters 13–14 will need graders for constraint adherence.
  • Cost of the instrument itself is absent: building the generator, adjudicating real history, running the ladder per mechanism — who pays, and does expense pressure select for cheap-to-test mechanisms?

The deeper territory

  • Goodhart pressure on the instrument. Once the ledger’s scenario list is public (it is — the spec names them), systems can be built to those scenarios rather than to memory. The real corpus is the defence, but it is smaller and hand-scored — precisely where Goodhart pressure concentrates least? Or most, since it gates publication claims? The chapter does not model the adversary.
  • Metric validity as the unmeasured assumption. Decision exactness, provenance precision, calibration — each assumes the ledger’s notion of correctness matches behavioural memory quality. But a system could score perfectly on ledger-derived answers while failing the Chapter 1 counterfactual (e.g., by memorising ledger-shaped patterns). The instrument measures ledger-reconstruction; the definition demands behaviour change. The gap between them is never examined.
  • The evaluator’s regress. Failure attribution includes “evaluator defect,” but who attributes attribution errors? Rubric-based human grading of downstream tasks (Ch 13, 20) reintroduces fluency judging through the back door unless rubrics are mechanical. The instrument needs a theory of its own trustworthiness, not just a defect category.
  • Frozen versus living. Freezing enables comparability; it also freezes in the author’s 2026 imagination of memory scenarios. A living benchmark updates, but then old runs incomparable. The chapter chooses frozen with versioning but never discusses the update policy: what evidence would force benchmark v0.2, and would v0.1 results survive it?
  • What the ledger cannot say. The hidden ledger is propositional and crisp: decisions, intervals, statuses. Real project memory includes graded, tacit, and contested content — half-formed consensus, authority that was never minuted, decisions everyone acted on but nobody recorded. If the generator can only plant crisp facts, the book’s memory is memory of the minuted, and the tacit half of real memory is out of experimental scope.

Concepts worth developing

The ledger-expressiveness bound

Idea. The set of phenomena the hidden ledger can represent upper-bounds the set of claims the book can test. Ledger gaps (graded commitment, entity merge/split, contested authority, tacit consensus) are blind spots of the whole investigation, not just missing fixtures.

Why it matters. It converts “we need more scenarios” into a precise question: what is the closure of the ledger ontology, and which memory phenomena fall outside it?

Connection to the current chapter. The chapter lists initial scenarios (§8 of the spec) as “at least” coverage, implying extensibility but never examining whether extension stays within the same ontology.

Broader implication. If some phenomena need ledger extensions, benchmark versioning must distinguish scenario additions (same ontology) from ontology changes (new ground truth kinds) — different compatibility consequences.

What remains unresolved. Which specific phenomena are actually outside the current ledger; whether tacit knowledge can be ledgerised at all.

Contamination as a temporal-validity problem

Idea. Treat benchmark contamination with Chapter 8’s own machinery: a query set has a validity interval (uncontaminated from release until absorbed into training data), and runs after supersession need a different interpretation. Contamination dating becomes validity-interval maintenance over the instrument itself.

Why it matters. It unifies two parts of the book and gives contamination handling a formal shape instead of a reporting convention.

Connection to the current chapter. The chapter already requires reporting publication dates versus cutoffs; this turns the report into maintained state with expiry semantics.

Broader implication. Held-out variants become “current belief” reserves; published sets become “historical” — still useful for regression, not for capability claims.

What remains unresolved. Whether training-data membership is inferable well enough to maintain such intervals honestly.

The metric-behaviour gap

Idea. Ledger-reconstruction scores (exactness, coverage, precision) and Chapter 1’s behavioural counterfactual can diverge in both directions: perfect reconstruction without behaviour change (the fluent labeller), behaviour change without faithful reconstruction (the lucky heuristic). The instrument needs bridge experiments showing its scores predict capstone behaviour.

Why it matters. Without the bridge, Chapters 2–19 could succeed on their own terms while the capstone fails, and nobody would know whether the layers or the metrics were at fault.

Connection to the current chapter. The chapter asserts per-question metrics follow from the behavioural definition but never demonstrates the entailment.

Broader implication. Suggests a standing validation study: correlate per-question scores with downstream task deltas once both exist — essentially a meta-experiment on the instrument.

What remains unresolved. How much correlation would validate the instrument; what to do if early questions predict nothing downstream.

Important distinctions

  • Mechanical ground truth versus ecological validity (and why neither implies the other).
  • Scenario addition versus ontology change in benchmark versioning.
  • Evaluator defect versus system defect (and who decides).
  • Contaminated versus expired query sets.
  • Ledger-reconstruction skill versus behavioural memory (the gap above).
  • Frozen comparability versus living coverage.

What mechanism would make this work?

The instrument is a pipeline: ledger → generator → artifacts (+ hidden labels) → system under test → outputs → ledger-derived scorers → per-question metrics + failure attribution → run manifest. Error entry points: generator artefacts (regularities), label ambiguity (two ledger-consistent answers), scorer bugs, evaluator drift, contamination, adjudicator disagreement. Reversibility: benchmark versioning (never silent mutation). What is missing as mechanism: the adjudication protocol for real history (sampling, agreement measurement, disagreement representation), the held-out rotation procedure, and the rubric-freezing procedure for human-graded tasks. v0.1 implements the task, output, scorer, manifest, and scorecard stages with canned fixtures standing in for the generator and query sets.

Connections to the rest of the book

  • Every experiment chapter (3–20) instantiates this chapter’s contract; the Type-A/Type-B pairs are the instruments’ cutting edge.
  • Chapter 20’s grader contract (“part of the fixture, not an afterthought”) is this chapter’s evaluator problem returning at full scale.
  • Chapter 13’s metric tension (resemblance metrics versus behaviour) is a special case of the metric-behaviour gap above.
  • The real corpus is the standing defence against every chapter’s Goodhart risk.

Beyond the current book

  • Measurement theory and psychometrics: validity (does the test measure the construct?), reliability (rerun stability) — the book reinvents both without naming them.
  • HELM (see Related work): scenario×metric matrices with explicit standardisation — the closest existing model for what this chapter attempts.
  • Causal inference: the frozen-run contract is essentially a protocol for controlled comparison; the counterfactual literature would sharpen it.
  • Software testing: frozen fixtures, regression suites, and oracle conditions are test-suite concepts; the harness is a test suite for memory.

The two deliverables and what v0.1 contains

The chapter now contracts two products. Component A, the Memory Measurement Instrument, determines whether a system remembers, what it remembers and forgets, whether retrieval, reconstruction, currency, evidence use, and abstention are correct, whether unsupported memories are fabricated, whether behaviour changed and improved, where failures occur, and how implementations compare. Component B, the memory system, is built in the remaining chapters. The instrument is never finished in Chapter 2: v0.1 provides the measurement model, task families, corpora design, baselines and controls, failure taxonomy, reproducibility contract, and the first deterministic scorers. Each later chapter adds tasks, metrics, and controls alongside the mechanism it earns. The capstone runs the complete instrument against the complete system.

Instrument status at the end of this pass: implemented — task/observation/manifest representations, mechanical scorers (source recall and precision, decision exactness, current-state and historical-state accuracy, supersession correctness, abstention correctness, unsupported-source detection), controls catalogue, scorecard rendering, 21 scorer tests passing, canned March-to-October demo runnable via python -m memory_measurement.demo from experiments/benchmark. Specified only — ranking metrics, support-chain validity, validity intervals, calibration, epistemic promotion errors, behavioural deltas, harm rates, cost accounting (all present as pending scorecard fields). Experiment pending — no frozen runs, no generator, no query sets, no book results.

Measurement dimensions (v0.1)

Preservation, retrieval (Recall@k, Precision@k, MRR, nDCG), reconstruction (proposal versus decision and cognates), provenance (evidence recall, evidence precision, support-chain correctness, unsupported-rationale rate), temporal correctness (current-state, historical-state, supersession, validity-interval accuracy, unresolved-conflict calibration), epistemic correctness (promotion errors such as hypothesis-reported-as-fact), groundedness (unsupported claim rate, fabricated source rate, incorrect provenance, confident unsupported recall), abstention, context selection (useful-memory precision, omission versus distraction), behavioural influence, behavioural utility, harmful memory as a first-class family (stale, irrelevant, contradictory, misleading, over-personalised, excessive, false-confidence-inducing), and cost (tokens, latency, storage, calls, context, money). Rates where lower is better (fabrication, harm) are marked as such on the scorecard so direction cannot be misread.

Hallucination-book reuse

The instrument inherits the companion volume’s evidence discipline with history substituted for retrieved documents: claim-evidence-edge granularity; available versus retrieved versus attributed evidence kept distinct; sensors as typed records rather than averaged scalars; answerability contracts with counterfactual add/remove-decisive-history probes; adversarial ladders (paraphrase-stable, negation-sensitive) with frozen development/held-out/regression suites; contamination accounting (taint-escape, exposure-before-containment, descendant counts); and every number carrying its provenance, grader, and stated limits. The standing distinction: hallucination asks whether a statement is supported; memory adds whether it was the correct historical or current state to bring forward.

Instrument evolution across the book

Decisions earn decision-reconstruction tests; provenance earns evidence-chain scoring; time earns supersession and current-state tests; open loops earn task-state metrics; context assembly earns relevance and distraction measurement; consolidation earns transfer and compression measurements; the capstone earns downstream behavioural measurement. Unearned instrumentation stays as specified interfaces and pending scorecard fields. The chapter-by-chapter measurement needs, artifact IDs, metric names, and failure classes gathered in this pass are recorded in the cross-book survey and govern what each later chapter must fund.

  • Liang et al., “Holistic Evaluation of Language Models” (arXiv 2022; Transactions on Machine Learning Research, 2023) — the scenario-by-metric evaluation matrix with standardised conditions is the nearest precedent for per-question scoring and frozen comparability, and its finding that models previously shared almost no evaluation scenarios motivates the frozen query sets. It does not solve ledger-style hidden ground truth or contamination expiry, which is where the book’s design goes further. Status: peer-reviewed. Landing page verified (arXiv 2211.09110; TMLR 08/2023).

Possible future claims

Already supportable

  • Demos without fixed cases, hidden labels, and reruns cannot distinguish remembering from paraphrase (argued from the four defects; conceptual, not empirical).

Plausible but needs development

  • The two-corpora design catches generator exploitation (mechanism described, no runs).
  • Per-question metrics measure distinct capabilities (predicted divergence, unmeasured).

Speculative

  • The ledger ontology suffices for all memory phenomena the book will need.
  • Real-corpus adjudication with preserved disagreement can anchor claims at reasonable cost.

Claims worth challenging

  • That hidden labels prevent circularity: the generator’s author and the ledger’s author are the same mind; systems built by that mind (or fine-tuned on its fixtures) may agree with the labeller for the wrong reasons. Hidden-from-the-system is not independent-of-the-author.
  • That scores must never merge: a decision-maker choosing between systems needs some total order; refusing all aggregation may abdicate the benchmark’s practical function. Is there an honest aggregate (e.g., per-family minima)?

Tensions and counterarguments

  • Rigour versus cost: the fuller the ladder, attribution, and adjudication, the fewer mechanisms get tested. The instrument may select for well-funded, simple claims.
  • Pre-registration versus discovery: Type-A/Type-B pairs constrain HARKing but may also blind the author to unpredicted patterns in the runs. Where is exploratory analysis allowed?
  • Synthetic control versus realism: every generator regularity is a potential exploit; every real-history messiness is a potential excuse. The no-merge rule keeps both visible but gives no procedure for trading them off when they disagree.

Examples and thought experiments

  • The author’s-own-system test: the book’s author builds a system that scores perfectly on the controlled corpus by pattern-matching generator templates. The real corpus catches it — but what if the author’s intuitions about “realistic” history shape both the generator and the adjudication rubric? Design the independence audit.
  • The ten-year test: benchmark v0.1 frozen in 2026, models of 2030 trained on its published queries. Which claims from v0.1 runs remain valid, and what exactly does the held-out rotation need to contain today to answer that?

Potential demonstrations or experiments

Question

Do per-question scores predict downstream behavioural deltas (the metric-behaviour bridge)?

Setup

Once any downstream task exists, correlate Q1–Q4 scores across systems with Q6 task deltas on matched scenarios.

Variable

System under test; held-out scenario families.

Observation

Correlation profile per question family.

Interpretation

Strong Q2/Q4→Q6 correlation validates the instrument’s decomposition; near-zero correlation indicts the metrics, not just the systems. Proposed; requires downstream tasks first.

Positive-control gating and config splits (added in cross-book review)

Two validity requirements developed in 13-concepts apply to the instrument generally and are recorded here so they govern all behavioural experiments: (1) every downstream experiment pre-registers a positive control (oracle-vs-naive must separate on these tasks) or a null verdict suspends rather than records; (2) policy/threshold parameters are tuned on development splits and evaluated on held-out tasks, with the split in the run manifest. Without these, Type-B verdicts confound “doesn’t matter” with “tasks can’t tell,” and Type-A verdicts risk tuning circularity.

Research questions this chapter creates

  • What scenario×metric evaluation designs exist beyond HELM, and which handle hidden ground truth?
  • What protocols exist for adjudicating temporal and intentional labels with measured agreement?
  • What is known about benchmark contamination half-lives for published query sets?

Architectural implications

  • The harness is itself a system with schemas (ledger, manifests, rubrics) that need versioning, migration, and invalidation — the book’s temporal machinery applies to its own instrument.
  • Scorers must be as tested as systems: scorer unit tests against hand-labelled edge cases belong in the frozen contract.

How would we know this works?

The instrument works if a system improvement that raises its scores reliably raises capstone behaviour, and if a template-exploiting system is caught by the real corpus. It fails if scores climb while behaviour stagnates — the Goodhart signature — or if no affordable run of the ladder is ever completed.

The chapter at its highest level

The ideal version would add: (1) a validity argument connecting ledger scores to the Chapter 1 counterfactual; (2) a complete adjudication and contamination-rotation protocol; (3) an explicit adversary model for benchmark gaming; (4) a benchmark-versioning policy distinguishing scenario, ontology, and scorer changes; (5) cost accounting for the instrument itself. The current version designs the machine; the ideal version would also prove the machine measures what it claims.

Discussion

Start here

  • What phenomena, if any, can the current ledger ontology not plant — and do those blind spots bound everything the book can conclude?
  • If a system scores perfectly on ledger reconstruction but fails the Chapter 1 counterfactual, which verdict stands — and what does that say about the instrument’s validity?
  • Who adjudicates the real corpus, with what agreement, and what happens to claims when adjudicators disagree about validity intervals?

Push the idea further

  • Model the adversary: if the scenario list is public, what does a system optimised for the benchmark without memory look like, and which control catches it first?
  • After five years of published query sets, what remains valid in v0.1 runs — and does the contamination plan as written actually answer that?
  • Should the book’s temporal-validity machinery be applied to its own query sets (validity intervals for uncontaminated status)?

Decisions we need to make

  • Whether any honest aggregate across question families is permitted, or the no-merge rule is absolute.
  • What forces benchmark v0.2, and whether v0.1 results survive it.
  • Whether rubric-graded human evaluation is allowed in the harness, and under what freezing rules.

Claims worth attacking

  • “Hidden labels prevent circular measurement.” Counter: author-shaped generator plus author-shaped adjudication can agree with each other for non-memory reasons.
  • “Scores must never merge.” Counter: decision-makers need total orders; minimax or per-family-minimum aggregation may be both honest and useful.

New ideas worth exploring

  • The metric-behaviour bridge study as a standing meta-experiment.
  • Ledger expressiveness auditing: systematically enumerating what the ontology cannot plant.
  • The harness as a temporal system: manifests, rubrics, and query sets under validity-interval maintenance.

Explain this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Apply this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Chapter 1 defined memory behaviourally: the past must change what the system does now, ideally improving it. This chapter defines how that claim is measured. The distinction matters from the first paragraph, because the word “instrument” will recur throughout the book and it never means the memory architecture. The memory system is one product of this book. The Memory Measurement Instrument — the apparatus used to determine whether that system remembers — is the other.

The book therefore makes an early promise with two deliverables. By the capstone, the reader should possess a memory system whose mechanisms were each earned by measured failure, and a measurement instrument capable of exposing the next failure. Neither is trustworthy without the other. We will not merely build memory. We will build the instrument required to determine whether that memory works.

The instrument is not finished in this chapter. What follows establishes version 0.1: the measurement model, the task families, the corpora, the baselines and controls, the failure taxonomy, the reproducibility contract, and the first runnable scorers. Later chapters extend it as new capabilities appear — decisions earn decision tests, provenance earns evidence-chain scoring, time earns supersession tests — until the capstone instrument can evaluate the complete system. The two products develop together: the memory system becomes more capable, and the instrument becomes more capable of exposing failure.

The question

The chapter revolves around a single question: how do we know whether an AI memory is any good? That question separates into four, and the instrument must answer all of them.

What do we measure? Which observable properties constitute memory quality — preservation, retrieval, reconstruction, provenance, temporal correctness, epistemic discipline, groundedness, abstention, selection, behavioural influence, utility, harm, cost?

How do we measure them? Which tasks, fixtures, controls, and metrics expose those properties without also rewarding fluency, lexical overlap, or benchmark exploitation?

How do we score them? What counts as success, partial success, failure, abstention, hallucination, regression, or harmful memory — and at what granularity?

How do we know the measurement itself is valid? This final question is the one most benchmarks skip. An instrument whose scores rise while downstream behaviour stagnates is measuring something, but not memory. Validity must itself be tested, and the chapter returns to it near the end.

Why demos are not evidence

A typical memory demo proceeds like this: the builder loads an interesting history, asks a question whose answer is known, and shows the system’s fluent response. The audience nods. Nothing about this procedure distinguishes remembering from paraphrase.

Four specific defects make demos unreliable as evidence.

First, the builder selects the case. Cases where the system happens to work are shown; cases where it confuses a proposal with a decision are not. Selection bias is invisible to the audience.

Second, the query is asked once, with a convenient phrasing. Whether the system survives paraphrase, temporal reframing (“what did we use in April?” versus “what should new code target?”), or a query that resembles a rejected idea is never tested.

Third, the answer is judged by fluency. A response that names the right topic, cites a real session, and sounds confident passes inspection even when it reports a superseded decision as current or attributes the decision to the wrong reason.

Fourth, nothing is rerun. When the next mechanism is added, there is no fixed set of old cases to check for regressions. Improvements are claimed; breakage is discovered by users.

The instrument exists to remove all four degrees of freedom at once: the corpus and queries are fixed before the system is built, the labels are hidden from the system, the metrics are defined per question, and every mechanism reruns the same cases.

What exactly are we trying to measure?

Chapter 1 ended with a ladder — preserved, retrievable, used, changed, improved — and with the warning that an item can pass the early rungs while failing the later ones. Chapter 2 converts that warning into a measurement model. The v0.1 dimensions below are a starting taxonomy, not a final one; later chapters may split or merge them as failures demand.

Preservation. Did the relevant historical information survive? This is the lowest rung and the easiest to verify mechanically.

Retrieval. Can the correct evidence be located? Standard ranking quantities apply here as established background: recall at k, precision at k, MRR, and nDCG where appropriate. [Primary sources for these definitions: citation TODO.]

Reconstruction. Can the system infer what actually happened — proposal versus decision, accepted versus rejected option, conclusion versus discussion? Retrieval success does not imply reconstruction success, and the book predicts the two will diverge.

Book hypothesis. Decision reconstruction will fail on a substantial class of cases even when retrieval of the relevant discussion remains strong, because the retrieved passages do not distinguish proposals from outcomes.

Provenance. Can the system identify why the conclusion exists? Measurements include evidence recall, evidence precision, support-chain correctness, and the rate of unsupported rationales. A plausible rationale with incorrect provenance scores as a failure.

Temporal correctness. Can the system distinguish what was true then from what is true now? Measurements include current-state accuracy, historical-state accuracy, supersession correctness, and validity-interval correctness.

Epistemic correctness. Can the system preserve how something is known — evidence, observation, inference, hypothesis, speculation, unresolved conflict, rejected idea, superseded belief? The characteristic error is promotion: a hypothesis reported as fact, a superseded belief reported as current.

Groundedness. Does the output introduce historical claims unsupported by the available record — fabricated sources, incorrect provenance, confident recall of events that never occurred?

Abstention. When the history does not determine an answer, does the system say so? Refusing to manufacture an answer is part of memory quality.

Context selection. When memory must fit a bounded context, were the useful memories selected and the harmful ones excluded? Measurements separate omission of critical memories from admission of stale, irrelevant, or contradictory ones.

Behavioural influence and utility. Did the retained past change what the system did, and did the change improve the outcome? These are the Chapter 1 counterfactual made runnable, and the hardest rung to instrument.

Harmful memory. Did memory make the system worse — through stale, irrelevant, contradictory, or misleading recall, excessive context, or false confidence induced by remembered material? Harm deserves first-class measurement, not an appendix.

Cost. Tokens, latency, storage, model calls, context usage. Memory quality is not independent of resource usage, and a system that remembers well at unbounded cost has shown less than it appears to.

No single number spans these dimensions. A system can show excellent retrieval with poor temporal memory, or excellent provenance with terrible behavioural utility, and an aggregate would hide both facts. The instrument reports a scorecard; any aggregate stays secondary and optional.

A task is not a metric

Much evaluation confusion comes from collapsing five distinct things into “a memory question.” The instrument keeps them separate:

  • Task. What event store should the new service use?
  • Ground truth. The hidden ledger establishes PostgreSQL as the current target, decided in adr-007, superseding SQLite.
  • Evidence. The ledger’s supporting entries: the contention experiment, the incident report, the decision record.
  • Expected behaviour. Use PostgreSQL and optionally explain the relevant reason.
  • Measurements. Correct current decision, correct evidence, no superseded recommendation, no fabricated rationale, appropriate use of memory.
  • Scores. Individual metric outputs, each traceable to the above.

This separation is enforced in code. A measurement task carries the prompt and a reference to the history the system may see, while every expected_* field is evaluator ground truth the system must never receive:

task = MemoryTask(
    task_id="decision-event-store",
    family="decision",
    prompt="What should new event-store services use?",
    history_ref="controlled-v0.1",
    expected_sources=("adr-007",),
    expected_state="PostgreSQL",
    superseded_options=("SQLite",),
)

A task family further specifies its input shape, ground-truth shape, expected evidence, allowed uncertainty, expected behaviour, metrics, and failure labels. The book’s six questions — locate, decide, justify, track truth through time, track unfinished work, select what matters now — are the initial families. They are measurement task families, not definitions of memory, and later chapters add families as new capabilities arrive.

What prior work measures

The papers already cited by this book were inspected for one question each: how did they know their system remembered better? The full research matrix lives outside the published chapters; what follows is what the instrument borrows and what it refuses.

LongMemEval separates information extraction, multi-session reasoning, temporal reasoning, knowledge updating, and abstention, with evidence-position labels in long distractor histories. The instrument borrows the ability split and the abstention task, and refuses judge-only scoring: every LongMemEval correctness verdict comes from a single proprietary model judge, which makes the judge a single point of validity failure.

LoCoMo evaluates very long conversational histories with adversarial items the system should resist, and compares retrieval over raw dialogues, observations, and summaries. The instrument borrows adversarial abstention items and the observation-level retrieval comparison, and refuses lexical overlap as a sufficient measure: word-match scores cannot separate recall from understanding.

LongBench contributes two controls the instrument adopts directly. A no-context condition separates what the model already knew from what the history supplied, and compression-versus-capability comparisons test whether a smaller representation preserves what matters. LongBench is otherwise refused as a memory instrument: it measures long-context use, not updating, abstention, or behaviour change from history.

AgentBench evaluates agents through interaction rather than prose, with a finish-reason taxonomy that distinguishes genuine completion from context-limit, invalid-action, and task-limit endings. The instrument borrows interactive evaluation and finish reasons for its later behavioural families, and refuses cohort-dependent aggregates and truncated histories that handicap long-horizon behaviour.

The memory systems of Chapter 1 supply a different lesson. MemGPT and MemoryBank both demonstrate that small samples and circular evaluation — testing on histories produced by the same model family that answers — flatter the system. Generative Agents shows the value of ablations that remove memory, reflection, and planning separately, so each component earns its keep. The instrument takes from all three the ablation discipline and the suspicion of self-graded histories, while insisting on what none of them provides: verifiable penalties for false memory alongside rewards for true recall.

Prior work reports (not book results): benchmark-scale memory evaluation currently leans on model judges, lexical overlap, and short histories. The instrument is designed so that each of those shortcuts has a control that exposes it.

Known truth and real history

The benchmark ingests two kinds of history because no single corpus answers both questions the book needs answered: does the mechanism work when ground truth is known exactly, and does it still work when history was not constructed for the test?

The controlled corpus: mechanical ground truth

A generator produces synthetic project histories from a hidden canonical ledger. The ledger is authoritative: it records which events occurred, which proposals became decisions, which facts held during which intervals, what superseded what, which evidence supported which conclusion, and which tasks remained open. A rendering step then turns the ledger into ordinary project artifacts: session transcripts, commits and commit messages, document revisions and diffs, issues and task updates, command transcripts, decision records, experiment notes.

The memory system sees only the artifacts. The evaluator sees the artifacts plus the ledger, and derives expected answers mechanically from the ledger. That separation is the core control of the whole benchmark. If the system could read the ledger, or if labels were inferred by a model’s judgement over the rendered text, the measurement would be circular: the test would reward systems that agree with the labeller rather than systems that reconstruct what happened.

To make the confusion impossible, the book adopts explicit terminology. The hidden representation is the Evaluator Ground-Truth Ledger: evaluation ground truth, not system architecture. The system under test never receives this representation. It receives only the rendered project history. Nothing about the ledger’s schema — event, decision, validity interval, supersession, contradiction, task, status, source — prescribes how a memory system should represent anything.

A minimal ledger entry, shown here as an illustration of the evaluation representation, carries the fields the spec requires. The canonical decision entry, simplified for illustration, is evt-205:

event_id: evt-205
kind: decision
topic: event-store target
content: New event-store work should target PostgreSQL.
actors: [m.okafor, j.lindqvist]
decided_at: 2024-07-11
valid_from: 2024-07-11
valid_until: null
supersedes: [evt-201]          # the earlier SQLite decision
supported_by: [evt-202, evt-203, evt-204]  # contention, benchmark, incident
source: [adr-007]
status: current

The ledger keeps production state separate from decisions. The matching production facts are fact-301 (SQLite runs in production, effective_at: 2024-03-04, valid until 2024-07-22, superseded) and fact-302 (PostgreSQL runs in production, effective_at: 2024-07-22, current, superseding fact-301). “What should new code target?” and “What runs in production?” therefore resolve against different records — the 11 July decision and the 22 July production state respectively.

The point of enumerating fields such as entity, claim, event, proposal, decision, rationale, validity interval, supersession, contradiction, task, status, and source artifact in the spec is not to dictate the memory system’s internal schema. It is to guarantee that the generator can plant the situations the book needs: discussion followed by decision, proposal followed by rejection, decision followed by reversal, fact followed by superseding fact, task followed by completion or abandonment, similar conversations with different outcomes, contradictory statements with and without resolution, conclusions that require combining multiple sources, and summaries that repeat a stale truth.

The real corpus: ecological validation

The second corpus is a genuine long-running project history, used with a smaller manually adjudicated query set. Every answer retains evidence references, and where adjudicators disagree, the disagreement is preserved rather than resolved by fiat.

The real corpus cannot provide mechanical ground truth at the same scale. Manual adjudication is expensive, and real history rarely contains clean labels for validity intervals or supersession. What it provides instead is resistance to generator artefacts. Any synthetic generator bakes in assumptions: how decisions are phrased, how far apart discussion and decision appear, how explicit timestamps are, how much vocabulary overlaps between related events. A mechanism that exploits those regularities will look strong on the controlled corpus and fail on real history. The real corpus exists to catch exactly that.

The two scores are never merged into one number. A mechanism earns its place by improving the controlled measurement for its intended failure class without unacceptable regressions elsewhere, and by surviving contact with the real corpus. Either result alone is insufficient.

Baselines and controls

Every experiment includes the applicable subset of the spec’s baseline ladder: no memory at all, lexical retrieval, embedding top-k, embedding plus reranking, the best system so far, the candidate system with the new mechanism, and an ablation with the mechanism removed or neutralized. The ablation matters more than it may seem. Without it, an improvement can be attributed to the mechanism when it actually came from a changed prompt, a larger context budget, or a different chunking policy that rode along with the patch.

The ladder needs adversaries as well as baselines. A random-history control, matched for size, checks whether any history helps or the right history helps. A scrambled-history control preserves token quantity and formatting while destroying content, separating content-caused improvement from longer-context effects. An oracle-memory condition supplies exactly the memory the task needs, giving an approximate ceiling for selection independent of retrieval. Stale-memory and distractor-memory adversaries inject plausible-but-superseded or relevant-looking-but-wrong material to test whether the system prefers current truth over familiar text.

The behavioural claim itself is comparative. Conceptually, every memory verdict has the shape:

without_memory = run(task_set, memory=None)      # schematic
with_memory = run(task_set, memory=memory_system)  # schematic

delta = score(with_memory) - score(without_memory)

The actual frozen-run implementation must control far more than these three lines show: corpus version and generator seed, task-set version, model identity, prompts, embedding and reranking models, chunking policy, retrieval and context budgets, memory configuration, code commit, grader identity and version, and scorer version. Any change to a frozen variable creates a new run. This is what makes “rerun the same cases” a promise rather than a slogan, and the run manifest records all of it so that any reported experiment is reconstructable.

Failure attribution uses the spec’s categories: ingestion, encoding, storage, retrieval, ranking, state and temporal reasoning, context assembly, downstream reasoning, and evaluator or label defect. The last category is easily overlooked and genuinely important. When the harness itself is wrong — a generator emitted an ambiguous timestamp, a query admits two ledger-consistent answers — the fix belongs to the benchmark, versioned as a new benchmark release, not silently folded into a system improvement.

What counts as failure

A single failed answer never produces a bare failure verdict. The instrument attributes each failure to the layer it indicts, because measurement should tell the experimenter which layer to repair. The v0.1 taxonomy distinguishes, among others, material never stored from material never retrieved, wrong retrieval from correct-evidence-wrong-interpretation, proposals reported as decisions from rejected options reported as accepted, stale state from missed supersession, missing provenance from false provenance, unsupported claims from failed abstention, omitted context from distracting context, and correct memory retrieved but not used from memory that actively harmed behaviour — plus evaluator defect and ambiguous ground truth for cases where the harness, not the system, is at fault.

Two scorers show the granularity. Provenance scoring separates what was found from what was relevant:

observations = score_task(task, output, history_ids)
# source_recall:    fraction of the ledger's required evidence retrieved
# source_precision: fraction of the retrieved material that was relevant
# unsupported_source_rate: fraction of cited sources absent from history

Temporal scoring keeps the two directions of truth apart. A historically accurate answer to a current-state question is stale, not partially correct:

# temporal_mode="current":    matches expected_current, else STALE_STATE
# temporal_mode="historical": matches expected_historical, else failure

A system that answers “SQLite” when asked what new code should target receives MISSED_SUPERSESSION, not a low similarity score. A system that answers “PostgreSQL has always been the choice” when asked about March receives a historical-state failure even though the sentence names the right technology. The failure name is the diagnosis.

Chapter 8 adopts and extends this scoring without changing its semantics: temporal-role accuracy (could the evidence have played the attributed role), order-sensitive change versus order-invariant stability (does the answer move exactly when reordering changes meaning), late-arrival robustness (actual versus known standpoints), future-effective accuracy (current versus planned across the effective boundary), sequence-gap detection, and projection rebuild equivalence. Each is reported per dimension; no aggregate temporal score is defined.

Chapter 9 extends the Question 5 family the same way: status accuracy over matched trajectories, cross-artifact and side-effect closure accuracy, open-loop precision and recall at matched mention sets, false-open and false-closed rates, stale-open and re-verification repair rates, bitemporal and historical status accuracy, gap calibration, footprint coverage, and maintenance/derivation agreement. Uncertainty is scored through calibrated classes (corroborated open, incomplete search, unknown), never through uncalibrated probabilities. No single open-loop score is defined.

Evidence and hallucination

A memory system saying “we decided PostgreSQL because of benchmark X” must be scoreable at several levels: did PostgreSQL actually become the decision, was benchmark X real, was it actually evidence for the decision, was necessary evidence omitted, was a rejected argument presented as rationale, is the claim current, was confidence appropriate? This is where the author’s companion volume on hallucination meets this book, and the instrument inherits its evidence discipline.

The Hallucination problem asks whether a statement is supported. The Memory problem adds whether this was the correct historical or current state to bring forward into the present. The instrument keeps both questions and never lets one answer stand in for the other.

Concretely reused machinery, adapted from retrieved documents to retained history:

  • Claim-edge granularity. Long responses score per claim-evidence edge, not per response. One supported claim does not launder an adjacent fabrication.
  • Three evidence sets kept distinct. History available in principle, history retrieved into context, and history cited for a specific claim are three different sets. Conflating them hides the most common failure: the right past was present and the system still argued from the wrong part of it.
  • Sensors are not verdicts. Scorer outputs stay typed records — entailment-style containment, trace status, source freshness — rather than averaged scalars. Policy decides what the numbers mean; measurement only reports them.
  • Answerability and abstention. Each task declares whether the history determines an answer. Counterfactual pairs probe the boundary: add the decisive history and the system should answer; remove it and the system should retrieve further or abstain.
  • Adversarial grading. Paraphrase must not move a score; negation, removal, role swaps, and temporal reversals must. Difficulty ladders run from random mismatch to structural inversion.
  • Contamination accounting. Taint-escape rates, exposure before containment, and descendant counts per root apply to benchmark material as much as to stored claims: a leaked query set is a contaminated instrument.
  • Every number with its provenance. The scorecard records where each figure was measured, with what grader, and what it does not establish — the Evidence Ledger pattern applied to the instrument itself.

Does the score mean anything?

An instrument that cannot be falsified is a ritual, not a measurement. The book therefore plans, from this chapter, the study that could indict its own benchmark: the metric-behaviour bridge. For several memory systems, per-question scores on the locate-through-temporal families are correlated with downstream behavioural improvement on matched tasks. If decision accuracy, current-state accuracy, and provenance precision rise while downstream behaviour stagnates, the verdict falls on the metrics, not the systems, and the instrument needs revision. That outcome is explicitly allowed. The benchmark is not protected from falsification.

Three further validity requirements govern every behavioural experiment the book will run. First, a positive control: an oracle-memory condition must separate from the naive baseline on the task set, or a null verdict suspends rather than records — without this, “no effect” confounds tasks that cannot tell with systems that do not differ. Second, selection and threshold parameters tune on development splits and evaluate on held-out tasks, with the split recorded in the manifest. Third, a fluent-summariser baseline — retrieval plus paraphrase with no behavioural use — must score poorly on every metric that claims to measure memory rather than text. Any metric it passes is invalid by construction.

Build it: the v0.1 implementation

Implemented, experiment pending (at v0.1 introduction). The code below exists and is tested; at the time this section was written no frozen runs had been produced with it. Later chapters’ runs exercise these scorers — see the instrument-state note above — but the full generator-backed benchmark the section describes remains pending. Nothing in this section is a book result.

The instrument lives in experiments/benchmark/memory_measurement/, alongside the experiment slots it will eventually execute. The current modules:

memory_measurement/
    tasks.py         # MemoryTask, HistoryItem, SystemOutput
    observations.py  # MemoryObservation, failure taxonomy
    scorers.py       # deterministic mechanical scorers
    controls.py      # baseline/adversarial conditions, frozen variables
    manifests.py     # RunManifest with hash-chained frozen variables
    reports.py       # scorecard rendering with pending-field labels
    demo.py          # runnable March-to-October demonstration
    tests/           # scorer tests: every branch covered

Task, observation, and manifest representations are implemented. Deterministic scorers cover source recall and precision, decision exactness, current-state and historical-state accuracy, supersession correctness, abstention correctness, and unsupported-source detection — each emitting typed observations with failure attribution. Ranking metrics, support-chain validity, validity intervals, calibration, epistemic promotion errors, behavioural deltas, harm rates, and cost accounting are specified interfaces with pending scorecard fields, earned by later chapters. Model-judged scoring is represented but unimplemented: where judgement is unavoidable, grader identity, version, and configuration will live in the manifest, never blurred into mechanical scores.

The demonstration runs six tasks over a March-to-October event-store history — locate the discussion, name the current target, justify it, report March truth, report current truth, and abstain where no cache decision exists — against three canned systems. From experiments/benchmark:

python -m memory_measurement.demo

A good system scores cleanly with no attributed failures. A stale system accumulates missed supersessions, stale-state errors, failed abstention, and a fabricated source. A fluent-but-wrong system names the right technology for today while failing March truth, citing a benchmark identifier that appears nowhere in history. The deltas separate all three, which is the minimum any instrument must do before it meets a real model.

A run eventually reports a scorecard resembling the demo output: per-metric values grouped by family, pending labels where v0.1 cannot yet measure, manifest hash for reconstructability, and failure attribution listing every task and failure class. Per-question results always precede any aggregate, and no aggregate is computed by the instrument itself.

Controls: what the harness must prevent

The chapter’s least glamorous section is its most load-bearing. A benchmark that leaks answers measures the leak, not the memory. The spec implies, and the harness notes under experiments/ implement, at least the following controls.

Answer leakage. The expected answer, or a paraphrase of it, must not appear in the rendered artifacts except where the ledger intends it. In particular, decision records that restate the ledger verbatim would let a system score on decision questions by pure extraction. The generator must render decisions the way projects do: sometimes crisp, sometimes buried in a session, sometimes split across artifacts.

Timestamp leakage. If every decision carries an explicit ISO timestamp in a fixed template position while discussion never does, a system can learn “prefer timestamped passages” instead of learning authority. Timestamps must be distributed the way real projects distribute them: present in commits, inconsistent in sessions, occasionally absent or relative (“last Thursday”).

Template leakage. If each scenario renders from one template with fixed phrasing, systems learn the template. The generator needs multiple surface realizations per scenario type and, where feasible, paraphrase passes that preserve ledger semantics while varying wording.

Lexical shortcuts. If the query shares distinctive vocabulary with exactly one artifact, retrieval succeeds without understanding. The initial scenarios deliberately include similar discussions with different outcomes and shared vocabulary, so that lexical overlap is necessary but not sufficient.

Query duplication. Queries must not repeat ledger labels verbatim in ways that let the system match strings rather than reconstruct state. Query phrasing is drawn from a separate pool from ledger content.

Evaluator privilege. The evaluator uses the ledger; the system must never see it, including indirectly through file paths, generation metadata, or deterministic ordering that correlates with scenario type. Frozen fixtures should be audited for such channels before any run counts.

Memorized patterns. Because the harness will eventually run against models pretrained on public text, frozen query sets must be treated as contaminated once published: later runs report the publication date of the query set relative to model training cutoffs, and unreleased held-out variants are kept for confirmation runs. [Citation TODO for current guidance on benchmark contamination.]

Experiment pending

At the time of writing, the generator, frozen fixtures, query sets, and harness described above do not yet exist as runnable artifacts beyond the v0.1 scorer package. The normative contract is spec/benchmark-v0.1.md; the intended directory layout is experiments/benchmark/README.md. Before any chapter can report a book result, the harness must produce at least: a versioned controlled corpus with hidden-ledger labelling, a versioned query set covering the early questions, frozen run manifests for the baseline ladder, and a failure-attribution procedure. A result of type A — strong locate recall with weak decision accuracy on matched cases — would support the book’s central progression. A result of type B — decision accuracy tracking locate recall closely across scenario types — would instead suggest that retrieval quality dominates and that later representational machinery needs stronger justification.

Instrument state after Chapters 3–7 (current note; the pending language above is preserved as the v0.1 historical record). The v0.1 scorers have since been exercised by frozen development runs: Chapter 3 ladder and reader comparisons, the Chapter 4 GraphRAG comparison (ch4-20260919T205622Z with Basic/Local/Global conditions plus an adjudication addendum), the Chapter 5 associative-retrieval suite (ch5-20260919-e5, retrieval-level only), the Chapter 6 routing matrix (ch6-20260919-nexus, 8 of 20 tasks, DRIFT unmeasured), and the Chapter 7 lineage suite (ch7-20260919-e7, fixture-level, zero model calls). Each run carries known limitations documented in its chapter; the full controlled-world generator, the real-corpus adjudication, and the metric-behaviour bridge remain open obligations. Nothing in this chapter’s pending section should be read as claiming those runs do not exist.

A living instrument

The v0.1 instrument is deliberately incomplete. Each later part of the book earns new instrumentation alongside new memory machinery:

Chapter introduces decisions
→ instrument gains decision-reconstruction tests

Chapter introduces provenance
→ instrument gains evidence-chain scoring

Chapter introduces temporal memory
→ instrument gains supersession and current-state tests

Chapter introduces open loops
→ instrument gains task-state metrics

Chapter introduces context assembly
→ instrument gains relevance and distraction measurement

Chapter introduces consolidation
→ instrument gains transfer and compression measurements

Capstone
→ instrument measures downstream behavioural improvement

If a planned component depends on machinery not yet introduced, the instrument carries the interface and an experiment slot, and the scorecard marks the field pending. That discipline is visible in the demo output already: a dozen pending fields, each one a promise a later chapter must either fund with a mechanism or delete honestly.

What remains unsolved

The contract is written and the first scorers run, but the generator, frozen fixtures, full query sets, and model-graded evaluation do not yet exist. The metric-behaviour bridge is a plan, not a plot. Adjudication protocols for the real corpus, held-out rotation procedures, contamination half-lives, and the cost of running the full ladder per mechanism are all open. The instrument measures what v0.1 can express; everything else is labelled pending until some chapter earns it.

The reader should now possess the beginning of one of the book’s two systems and almost nothing worth calling an advanced memory system. That is deliberate. We now have a way to expose memory failures. We can finally build the simplest possible memory system and see where it breaks.

That is the next chapter’s job — and the scientific rhythm of the whole book: define, measure, build, fail, diagnose, improve, measure again. By the capstone, the payoff takes a fixed operational shape: attach or implement a memory system, define the tasks the application needs memory for, run those tasks through the instrument, inspect where memory succeeds and fails, change the architecture, rerun the same instrument, and determine whether the change actually helped. The book teaches not only a memory architecture, but how to measure any memory architecture after the book is finished.

What prior work adds to the instrument

LongMemEval separates information extraction, multi-session reasoning, temporal reasoning, knowledge updating, and abstention. LoCoMo distributes evidence across very long conversations and probes resistance to false premises. LongBench shows that long-context ability itself spans different task families and that no-context controls separate prior knowledge from supplied history, while AgentBench evaluates agents through interaction with explicit finish reasons rather than prose alone.

These designs reinforce two choices in the benchmark. First, the early questions need separate scores: success at locating a passage does not certify decision reconstruction, temporal maintenance, or action. Second, errors must be attributed to a stage — indexing, retrieval, reading, maintained state, assembly, or action — because an end score cannot identify the correct repair.

They also strengthen abstention as a first-class outcome. When the history does not determine an answer, refusing to manufacture one is part of memory quality. The controlled ledger makes that mechanically scoreable; the separately adjudicated real corpus tests whether the distinction survives histories not written for evaluation.

Research foundations

Long-horizon memory benchmarks support the decision to separate capabilities rather than score memory through a single fluent answer. LongMemEval isolates information extraction, multi-session reasoning, temporal reasoning, knowledge updating, and abstention; LoCoMo evaluates very long conversational histories; LongBench and AgentBench illustrate the broader need for fixed tasks, controlled evaluation, and per-capability reporting. They inform the instrument, not the outcomes of this book’s still-pending experiments.

References