# The Memory Contamination Problem
# Book: Hallucination From First Principles
# Chapter URL: https://programmer.ie/books/hallucination-from-first-principles/14-chapter/
# Prose fingerprint (sha256, normalized): bdd434f0a70437fa099167cb8c9e38db0cf0ba8f327d7510efc746a8d1b0c6d4
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

Chapter 13 ended with a strict recovery invariant:
repair → new candidate state → re-measure → re-authorize and with one additional rule:
candidate in HOLD → do not persist as trusted factual memory That second rule changes the scale of the problem.
A hallucinated sentence displayed once is transient.
The same sentence written into persistent state can survive the conversation that created it.
It can then be:
retrieved summarized copied cited used as agent experience used to fill a user profile inserted into a vector index fed into another model used as a future repair source and eventually appear to the system as if it came from somewhere else.
The failure has changed form.
It is no longer merely:
model generated unsupported claim It has become:
system state contains unsupported influence That is the subject of this chapter.
Persistent memory is a future influence channel. In factual systems, it can become a future evidence channel.
A contamination failure across two sessions Consider the claim from Chapter 13:
Q3 revenue was approximately $46 million. The available evidence contains no Q3 revenue value.
The correct verification state is:
INSUFFICIENT_EVIDENCE Now imagine a naive memory writer that stores every model-generated statement that looks useful.
Session 1 candidate = "Q3 revenue was approximately $46 million." verification = INSUFFICIENT_EVIDENCE response commitment = HOLD A flawed writer nevertheless produces:
memory m1 content = "Q3 revenue was approximately $46 million." origin = MODEL_OUTPUT lifecycle = ACTIVE Session 2 The user asks:
What was Q3 revenue? A similarity retriever returns m1.
The prompt now contains:
Relevant memory: Q3 revenue was approximately $46 million. The generator answers:
Q3 revenue was approximately $46 million. No external source has entered the system.
The unsupported assertion has merely changed roles:
graph LR S1[SESSION 1 MODEL OUTPUT] --> PM[PERSISTENT MEMORY] PM --> S2[SESSION 2 RETRIEVED CONTEXT] S2 --> S2O[SESSION 2 MODEL OUTPUT] That role change can create a false appearance of evidence.
To make the admission failure concrete, I built a deterministic reference example.
Environment Python 3.13.5 standard library only no external model calls The observed output was:
FLAWED_STORE m1 origin=MODEL_OUTPUT verification=UNVERIFIED lifecycle=ACTIVE flawed factual retrieval: ['m1'] GOVERNED_STORE m1 origin=MODEL_OUTPUT verification=UNVERIFIED lifecycle=QUARANTINED m2 origin=EXTERNAL_SOURCE verification=VERIFIED lifecycle=ACTIVE m3 origin=MEMORY_DERIVED verification=UNVERIFIED lifecycle=QUARANTINED parents=['m1', 'm2'] factual-evidence retrieval: ['m2'] general recall: ['m2'] The important line is m3.
It is a derived summary built from one verified memory and one unverified memory.
The summary does not become verified merely because a new transformation rewrote the inputs into cleaner prose.
Transformation does not upgrade provenance.
The controller example proves that invariant under a deterministic structured oracle.
It does not establish the accuracy of real-world memory extraction, verification, or summarization.
Where we are The reliability architecture now has three distinct boundaries.
Response commitment candidate → measurement → policy → may this leave the system as an assertion? Memory admission memory candidate → admission policy → may this become persistent state, with which capabilities? Memory use stored memory → access + admissibility policy → may this influence this purpose? These boundaries are related.
They are not interchangeable.
A response that received:
commitment = PERMIT is only eligible for memory admission evaluation.
It is not automatically admitted.
Likewise, a memory that was legitimately persisted is not automatically admissible for every future task.
This gives the chapter’s first type discipline:
$$ \boxed{ \text{response PERMIT} \not\Rightarrow \text{memory ADMIT} \not\Rightarrow \text{admissible for every use} } $$ 1. Persistence changes the failure class Without persistence:
bad generation → bad response → session ends With persistence:
bad generation → stored state → future retrieval → future reasoning → future output → possible re-storage The first is primarily an output error.
The second is a state corruption process.
Persistent writable memory introduces:
persistence statefulness propagation A 2026 survey of long-term-memory security in LLM agents uses those properties to explain why cross-session writable memory creates a qualitatively different attack surface.[1]
The book’s concern is broader than adversarial attacks.
The same state can be contaminated by:
hallucinated facts stale facts misresolved identities incorrect summaries failed agent trajectories unverified user assertions bad retrieval results repair candidates that never passed policy The database cannot tell whether a false value came from:
a confused model or:
a malicious injection unless the system preserved origin and admission evidence.
So accidental contamination and adversarial poisoning meet at the same architectural boundary:
WRITE Their threat models differ.
Their need for governed writes does not.
2. Memory is typed state, not one epistemic object The word memory hides several very different objects.
interaction event user assertion user preference tool observation external-source claim model inference derived summary agent procedure failure experience These objects answer different questions.
Consider:
User said: "My manager is Alice." There are at least two possible memory propositions.
Assertion event At time t, the user stated: "My manager is Alice." The system may have directly observed that event.
Asserted world proposition manager(user) = Alice That proposition is still merely user-asserted unless another admissible source establishes it.
Therefore:
$$ \boxed{ \text{verified assertion event} \neq \text{verified asserted proposition} } $$The same applies to tools.
git status returned a clean working tree at 18:32 may be a verified runtime observation.
It does not establish:
the repository is clean forever. Memory should preserve what was actually observed, asserted, inferred, or derived.
3. Split origin, verification, time, lifecycle, and capabilities The first draft of this chapter used one broad trust enum containing values such as:
VERIFIED USER_ASSERTED TOOL_OBSERVED MODEL_DERIVED STALE REVOKED That mixes different dimensions.
A stronger memory record separates them.
Origin Where did this representation come from?
USER TOOL EXTERNAL_SOURCE MODEL_OUTPUT MEMORY_DERIVED Verification state What is established about the proposition?
VERIFIED UNVERIFIED INSUFFICIENT_EVIDENCE REFUTED CONFLICTING UNRESOLVED NOT_VERIFIABLE Temporal validity When is the proposition supposed to hold?
valid_from valid_to System time When did the system observe and store it?
observed_at recorded_at superseded_at Lifecycle What is its persistence state?
EPHEMERAL QUARANTINED PROVISIONAL ACTIVE ARCHIVED SUPERSEDED REVOKED NEEDS_REVALIDATION Capabilities For which purposes may it be consumed?
PERSONALIZATION CONVERSATIONAL_CONTEXT FACTUAL_EVIDENCE AUTONOMOUS_ACTION_JUSTIFICATION REPAIR_PLANNING REGRESSION_TEST PROCEDURE_IMITATION Now the system can ask different questions independently:
Where did this come from? Is its proposition established? Is it still current? May it still be retained? May it be used for this purpose? One enum should not answer all five.
4. Memory admission is its own policy decision A memory candidate should cross a write gate just as a generated answer crosses an acceptance gate.
A typed record might look conceptually like:
memory_candidate = { "memory_id": "mem_1842", "content": "Q3 revenue was approximately $46 million.", "memory_type": "FACTUAL_CLAIM", "origin_kind": "MODEL_OUTPUT", "verification_state": "INSUFFICIENT_EVIDENCE", "lifecycle_state": "EPHEMERAL", "parent_memory_ids": [], "support_claim_ids": [], "evidence_ids": ["filing_q1", "filing_q2"], "verification_record_id": "verify_1842", "requested_uses": ["FACTUAL_EVIDENCE"], "valid_from": None, "valid_to": None, "observed_at": "2026-08-30T18:39:00+01:00", "retention_scope": "project", } Admission then decides capabilities, not one global trusted bit.
For example:
admission_decision = { "persist": True, "lifecycle_state": "QUARANTINED", "allowed_uses": ["REGRESSION_TEST"], "forbidden_uses": [ "FACTUAL_EVIDENCE", "AUTONOMOUS_ACTION_JUSTIFICATION", ], "policy_version": "memory-admission-2026.08.30.3", } Recent work on Adaptive Memory Admission Control similarly treats long-term-memory admission as an explicit structured decision rather than automatic retention or purely opaque LLM judgment.[2]
One important qualification follows from our architecture:
Future utility and epistemic admissibility are different objectives.
A recurring model misconception may be extremely useful as a regression artifact while remaining forbidden as factual evidence.
5. Derivation carries provenance obligations Suppose:
m1 = unverified rumor m2 = verified Q2 revenue and a summarizer creates:
m3 = summary(m1, m2) It is tempting to assign one scalar trust score to each node and define:
$$ q(m_3) \le \min(q(m_1),q(m_2)). $$That captures one intuition but is too coarse for a general memory system.
USER_ASSERTED, TOOL_OBSERVED, VERIFIED, and MODEL_DERIVED are not naturally one total numerical order for every purpose.
A better rule is capability and lineage preserving.
For each factual claim in the child, record which parent claims are actually required to support it.
For example:
summary_claim = { "claim_id": "m3:c2", "content": "Q2 revenue was $43.8M.", "required_parent_claims": ["m2:c4"], "derivation": "SUMMARIZE", } If another summary claim depends on both m1 and m2, the unresolved provenance obligation from m1 survives into that claim.
So:
A derived claim carries the unresolved provenance obligations of every required support path until new admissible verification resolves them.
The entire document need not inherit taint from an unrelated parent that merely appeared in the summarizer context.
That is why claim-level lineage matters.
A derivation cannot create stronger factual evidence than its required sources contain.
6. Repetition is not corroboration Suppose the same unverified claim is stored in three forms:
m1: Q3 revenue was approximately $46M. m2: Revenue reached roughly $46M in Q3. m3: Third-quarter revenue was about forty-six million dollars. A vector retriever may return all three.
The generator sees:
three agreeing passages but lineage reveals:
cand_v1 ├── m1 ├── m2 └── m3 This is:
3 texts 1 evidential lineage But even root count is not enough.
Suppose the system sees:
news_site_A news_site_B blog_C All three may independently exist as memory roots while ultimately copying one press release or one original error.
Therefore:
$$ \text{root count} \neq \text{independent evidence count} $$The lineage model should support relationships such as:
DERIVED_FROM COPIED_FROM SYNDICATED_FROM SHARES_PRIMARY_SOURCE_WITH SOURCE_FAMILY At retrieval time, corroboration should be computed over independent admissible source families, not chunk count.
Repeated text is not independent evidence.
7. Derivation lineage and evidential support are different graphs A versioned derivation graph usually moves forward in time:
cand_v1 → mem_v1 → summary_v1 → cand_v2 → mem_v2 That graph can remain a DAG.
But the support graph asks another question:
Which proposition is being used to justify which proposition? A self-confirming system can have circular support even if its immutable derivation history is acyclic.
For example:
model invents X → stores X → later retrieves X → cites retrieved X as support for X Derivation history:
cand_v1 → mem_v1 → cand_v2 Support relation:
X is justified by a representation descended from X itself That is circular support.
The correct question is:
Where do the support paths terminate? If all admissible-looking support ultimately descends from the proposition being justified, no independent verification occurred.
A claim cannot verify itself by taking a trip through a vector database.
8. The read path needs access control before relevance and admissibility after relevance A reliability-aware memory read should not be:
query → top-k cosine similarity → prompt It needs at least three stages.
graph TD Q[QUERY + PURPOSE + CALLER] --> HA[HARD ACCESS / ELIGIBILITY FILTER] HA --> SR[SEMANTIC RETRIEVAL] SR --> EA[CONTEXTUAL / EVIDENTIAL ADMISSIBILITY] EA --> UC[USABLE CONTEXT] Memory retrieval is governed before and after similarity search, so a relevant memory is still excluded if it lacks the rights or evidential status required for the current use.
Pre-retrieval eligibility Exclude memories that should not even enter the retrieval candidate set because of:
tenant / user boundary sensitivity lifecycle state caller authorization hard purpose restriction legal deletion / retention state Semantic retrieval Rank eligible memories for relevance.
Post-retrieval admissibility Evaluate:
verification state freshness conflict scope source-family independence task risk requested purpose This matters operationally because an item can leak before final prompt construction if a reranker, log, or remote retrieval service sees it.
For a simple PostgreSQL + pgvector deployment, some of the first gate can be ordinary metadata filtering:
SELECT * FROM memory WHERE lifecycle_state = 'ACTIVE' AND allow_factual_evidence = TRUE AND tenant_id = $1 ORDER BY embedding <-> $2 LIMIT 10; pgvector supports nearest-neighbor queries with ordinary SQL WHERE filters, although approximate indexes require care because filtering can occur after index scanning and may affect recall.[10]
This is enough for some applications.
It is not enough for all of them.
Lineage, conflict, supersession, claim dependencies, and source-family relations are graph-shaped state. A graph-backed or hybrid graph-plus-vector architecture can represent those relations explicitly while vector search handles semantic candidate retrieval.
Microsoft’s GraphRAG is one concrete research implementation: its indexing pipeline extracts entities, relationships, claims, community structure, and embeddings.[11]
It should not be treated as a universal industry standard—the Microsoft repository itself describes the project as research-oriented and, as of 2026, largely in maintenance mode.[11]
The architectural point is broader:
vector search → relevance graph / metadata / policy → epistemic structure and admissibility 9. Context serialization should expose restrictions without dumping the database Rich memory records may contain:
lineage source families verification records validity intervals conflict edges policy versions The model usually does not need every field verbatim in its context window.
Dumping the entire memory schema into the prompt can add latency, token cost, and distraction.
A better read layer serializes the minimum decision-relevant metadata.
For an ordinary verified memory:
Q2 revenue was $43.8M. [source: filing_q2] For an anomalous memory:
<memory state="CONFLICTING" use="CONTEXT_ONLY"> Sources disagree about the current CEO. </memory> For a user assertion:
<memory origin="USER" verification="UNVERIFIED" use="PERSONALIZATION"> User stated that their manager is Alice. </memory> The full provenance record remains outside the model and can be dereferenced when policy or verification needs it.
Prompt context should expose semantics that affect model behavior, not mirror the entire persistence schema.
10. Factual memory should often point back to evidence A verified memory should not become authoritative merely because the system copied a fact into its own store.
Where possible, factual memory should act as an index into durable evidence.
Instead of storing only:
Q2 revenue = $43.8M verification = VERIFIED retain:
normalized claim verification state source ID source version exact support span evidence snapshot reference retrieval / verification time valid_from / valid_to Then a high-risk future action can use:
memory hit → dereference original evidence → verify source still admissible / current → use source as evidence This preserves a crucial distinction:
memory helps find evidence versus:
memory has become the evidence simply by being stored. 11. Summaries and compaction need claim-level preservation contracts Long-running systems summarize memory to control context size.
That creates a trust-laundering surface.
Suppose raw memory says:
Source A reports X. Source B disputes X. verification = CONFLICTING A bad summarizer writes:
X is true. Two failures occurred.
Epistemic elevation CONFLICTING → VERIFIED-looking assertion Lineage collapse A + B → summary with source identities lost So compaction should preserve, at claim level where relevant:
required parent claims verification state source families conflict state uncertainty validity interval scope A safe summary might instead say:
Sources disagree about X. and preserve links to both claims.
The contract is:
Compression may reduce tokens. It must not silently increase certainty or erase the lineage required to interpret the claim.
Recent 2026 work on MemGuard takes a related lifecycle approach by persisting verifier outputs and reusing them during memory admission, retrieval, conflict handling, summarization, and archival.[6]
12. Time and conflict need resolved semantics Persistent memory needs at least two time dimensions. This is standard bitemporal modelling — the valid-time / transaction-time split from Snodgrass, carried into SQL:2011 as system-versioned tables.[14]
Valid time When was the proposition true in the world?
valid_from valid_to System time When did the system learn and store it? (Bitemporal databases call this transaction time; SQL:2011 calls it system time.)
observed_at recorded_at superseded_at For example:
CEO = Bob valid_from = 2026-07-01 source_published = 2026-07-03 system_recorded = 2026-07-05 Now two questions can be answered differently:
Who was CEO on June 20? and:
What did the system believe on July 2? Conflict detection also needs resolved identity, relation, time, and scope.
These are not necessarily contradictory:
Alice was CEO in 2025. Bob was CEO in 2026. Nor are:
CEO of US parent = Alice CEO of European subsidiary = Bob So before creating:
m1 CONFLICTS_WITH m2 resolve:
entity relation scope validity interval Memory conflict is a structured relation, not merely text that appears opposite.
13. Revocation invalidates descendants; it does not blindly delete them Suppose memory m1 later becomes invalid.
And suppose m1 contributed to:
m3 summary m7 recommendation m9 profile entry m12 graph edge Changing only m1 is not enough.
But blindly revoking every descendant is also wrong.
A descendant may have independent support.
For example:
m3 depends on: m1 authoritative m2 If m1 is revoked, m2 may still establish the proposition.
Therefore:
$$ \text{revoke(parent)} \not\Rightarrow \text{revoke(all descendants)} $$The correct operation is dependency invalidation:
graph TD REV[REVOKE / CHANGE ROOT] --> FIND[find dependent claims] FIND --> MARK[mark NEEDS_REVALIDATION] MARK --> RECOMP[recompute support from remaining admissible roots] RECOMP --> STATE[ACTIVE / QUARANTINED / REVOKED] This is ordinary incremental-build invalidation applied to epistemic state.
The system also needs impact analysis for artifacts that have already escaped memory:
responses tool actions database writes reports notifications external decisions Some consequences cannot be rolled back by changing memory.
They may require:
correction notification compensating action reissue human escalation 14. Epistemic revocation and privacy erasure are different operations For epistemic correction, versioned history is often valuable:
old assertion → superseded / revoked → corrected version That supports replay and audit.
But personal-data governance can impose different requirements.
The GDPR’s Article 17 provides a right to erasure in specified circumstances and requires erasure without undue delay when applicable, subject to its conditions and exceptions.[12]
California’s CCPA likewise provides consumers a right to delete personal information collected from them, subject to exceptions.[13]
Therefore:
epistemic correction ≠ privacy erasure A privacy deletion workflow may need to:
remove or irreversibly anonymize content remove embeddings / replicas where required invalidate descendants that contain the data propagate deletion to caches and derived stores stop future retrieval while retaining only whatever non-content tombstone or audit metadata is lawfully necessary.
The exact legal obligation depends on jurisdiction, purpose, exemptions, and the application’s role; this chapter is not legal advice.
The systems point is narrow and important:
Do not make “we preserve all history forever” an invariant of the memory architecture.
The same lineage machinery used for epistemic rollback is also what makes targeted erasure technically possible.
15. Failure and experience memory need environment fingerprints Agents increasingly store:
plans procedures tool sequences successful trajectories failed trajectories These memories can influence behavior even when they are not factual propositions.
A failure should be stored as a failure:
failure_memory = { "memory_type": "FAILURE_EXPERIENCE", "outcome": "FAILED", "allowed_uses": ["REGRESSION_TEST", "REPAIR_PLANNING"], "forbidden_uses": ["SUCCESSFUL_PROCEDURE_IMITATION"], } A successful trajectory also needs conditions under which success was established:
repo commit tool versions OS/runtime permissions inputs preconditions postconditions verifier result Otherwise:
This command sequence solved the issue. can be retrieved into a completely different environment and treated as reusable policy.
MemoryGraft is important here because its attack surface is precisely successful-looking experience memory: poisoned trajectories can later be retrieved and imitated on semantically similar tasks.[4]
So procedural memory needs:
success + the environment and constraints under which success was established.
16. Accidental contamination and adversarial poisoning require overlapping but different controls The same write gate protects against both accidental and malicious persistence.
But the threat models differ.
Dimension Accidental contamination Adversarial contamination typical origin hallucination, stale fact, bad summary poisoned document, malicious memory write, sleeper trigger adversary absent controls or influences input/write path shared control admission, lineage, verification, use policy admission, lineage, verification, use policy extra emphasis quality, staleness, conflict origin authentication, write authorization, rate limiting, isolation RevPRAG studies poisoned RAG databases and detection of poisoned responses.[3]
MemoryGraft studies persistent compromise through poisoned experience retrieval.[4]
Hidden in Memory studies delayed poisoning in which adversarial external context induces fabricated persistent memories that later steer stateful assistants.[5]
These are security attacks.
The chapter’s broader reliability lesson is:
The memory store should not need to know whether bad state was malicious before it applies basic epistemic controls.
17. Evaluate contamination across the lifecycle A useful benchmark should be organized by where the failure enters or escapes.
Phase Representative metrics What they catch Write / admission bad-memory admission, false quarantine, verification coverage unsafe writes or excessive blocking Lineage claim-lineage completeness, source-family coverage, taint-escape rate provenance laundering Read polluted retrieval, inadmissible exposure, stale retrieval relevance overriding admissibility Use memory-induced false acceptance, wrong action, self-confirmation downstream harm from memory Maintenance revalidation completion, revocation correctness, rollback latency persistent stale/invalid state Utility useful recall, personalization retention, latency, storage cost safe-but-useless memory policy Two especially useful operational metrics are:
Taint-escape rate How often does a descendant with unresolved contaminated ancestry incorrectly become admissible as clean factual evidence? Exposure before containment How many retrievals, responses, or actions consume contaminated state before quarantine or revocation occurs? Also report:
descendant count per contaminated root maximum propagation depth post-revocation exposure false invalidation rate One memory-quality scalar would hide these failure modes.
Event-driven plus periodic maintenance Do not rely only on scheduled scans.
Trigger immediate invalidation when:
source revoked source updated parent memory revoked user correction arrives entity merge changes resolution policy changes schema changes Then run periodic scans for:
staleness poisoning patterns missed conflicts long-lived high-impact memories CAMS proposes a lifecycle security architecture containing guarded writes, provenance, temporal monitoring, graph analysis, periodic scanning, and retroactive quarantine.[7]
18. A production memory architecture The chapter’s canonical architecture now has one write gate and two read gates.
Write side graph TD US[USER / TOOL / SOURCE / MODEL] --> MC[MEMORY CANDIDATE] MC --> OVT[ORIGIN + VERIFICATION + TIME] OVT --> CL[CLAIM-LEVEL LINEAGE] CL --> AP[ADMISSION POLICY] AP --> PERS[PERSISTED WITH CAPABILITIES] The write gate turns memory creation into an authorization step: only candidates with origin, verification state, time, and lineage can receive durable use capabilities.
Read side graph TD Q[QUERY + PURPOSE + CALLER] --> AE[ACCESS ELIGIBILITY] AE --> RR[RELEVANCE RETRIEVAL] RR --> EA[EVIDENTIAL ADMISSIBILITY] EA --> CS[CONTEXT SERIALIZER] CS --> GA[GENERATION / ACTION] Every generated output that might be remembered returns to the write side.
This gives three different questions:
May we store it? May this caller retrieve it? May it be used for this purpose? Those questions should never be collapsed into:
is this vector in the database? Memory ledger Store enough metadata to replay how the memory acquired its current capabilities:
memory_id content / immutable content reference memory_type origin_kind verification_state verification_record_id valid_from / valid_to observed_at / recorded_at lifecycle_state parent claim IDs source-family IDs allowed uses forbidden uses admission_policy_version memory_schema_version verification_schema_version embedding_model_version normalizer_version entity_resolution_version evidence snapshot reference + hash Versioning the schema and resolvers matters because a later entity-resolution or policy change can alter what historical records mean.
19. Counterfactual memory tests The metamorphic discipline from earlier chapters applies directly.
Same content, different origin Hold text fixed:
$$ content(m_1)=content(m_2) $$but change:
EXTERNAL_SOURCE + VERIFIED → MODEL_OUTPUT + UNVERIFIED Expected:
factual admissibility may decrease. The system must respond to epistemic metadata, not only text.
Derivation laundering graph LR m1[verification=UNVERIFIED] --> m2[summarize] --> m3[summarize] --> m4[summarize] Summarization may change wording, but it must not launder an unverified memory into a fact with stronger evidential rights.
Expected:
m4 remains lineage-linked to m1 m4 does not acquire factual-evidence capability Source-family duplication Create five paraphrases and three URLs that all descend from one primary source.
Expected:
chunk count > source-family count corroboration uses source-family count Revocation Revoke one required root.
Expected:
descendants become NEEDS_REVALIDATION not blindly all REVOKED Staleness Advance the requested as-of date beyond valid_to.
Expected:
memory loses current-state authority. Response permission Change:
response commitment HOLD → PERMIT Expected:
HOLD cannot enter active factual memory PERMIT becomes eligible for admission policy PERMIT does not auto-admit 20. Using AI: extract memory candidates, do not certify them An LLM can help turn conversation into structured memory candidates.
A weak instruction is:
Remember the important facts from this conversation. That lets the model decide:
what is important what is factual what is persistent what is verified what may be reused A stronger instruction is:
Extract candidate memory units only. For each candidate return: - exact source span - proposition_vs_observation - memory type - normalized content - origin kind - temporal scope - entities involved - possible parent claims - requested future use Do not mark anything VERIFIED. Do not decide persistence. Do not remove uncertainty or conflict. For:
My manager is Alice. AI may propose two objects:
ASSERTION_EVENT "User stated that their manager is Alice." origin = USER verification = VERIFIED_OBSERVATION_OF_EVENT and:
WORLD_PROPOSITION "User's manager is Alice." origin = USER verification = UNVERIFIED The system—not the extraction model—decides admission and future capabilities.
A common extraction failure is:
User mentioned X → extractor marks X VERIFIED That should fail schema/policy validation.
The model observed an assertion.
It did not verify the proposition asserted.
What you should now be able to answer After this chapter, you should be able to explain:
Why persistent hallucination is a state-corruption problem rather than only an output error. Why origin, verification, temporal validity, lifecycle, and use capabilities must be separate fields. Why a verified assertion event is not the same thing as a verified world proposition. Why response authorization does not imply memory admission. Why transformation and repetition cannot manufacture independent evidence. Why claim-level lineage and source families matter for corroboration. Why derivation history and evidential support are different graphs. Why retrieval needs access filtering before similarity search and admissibility after it. Why factual memory should often dereference original evidence for high-risk use. How compaction, time, conflict, revocation, privacy erasure, and procedural memory change the governance problem. Exercises Exercise 1 — Build a typed memory gate Create four candidates:
verified tool observation user-stated preference model-generated factual inference failed repair candidate For each, assign:
origin verification state temporal scope lifecycle allowed uses Confirm that the model inference cannot become factual evidence merely through admission.
Exercise 2 — Test trust laundering Create:
m1 origin=MODEL_OUTPUT verification=UNVERIFIED Then execute three summarization steps:
m1 → m2 → m3 → m4 Write a unit test asserting:
m4 remains connected to m1 through claim-level lineage FACTUAL_EVIDENCE not in m4.allowed_uses Then add a new admissible external source that independently verifies the proposition and test that policy may explicitly upgrade the new version.
Exercise 3 — Test source-family corroboration Create five stored chunks from three URLs, all syndicated from one primary source.
Verify:
chunk_count = 5 url_count = 3 independent_source_family_count = 1 Exercise 4 — Revoke and revalidate Build:
m1 → m3 → m7 m2 → m3 where m2 independently supports m3.
Revoke m1.
Confirm that:
m3 → NEEDS_REVALIDATION and is restored if m2 is sufficient, rather than blindly revoked.
Exercise 5 — Relevance versus admissibility Create a query for which:
highest similarity = QUARANTINED / UNVERIFIED second similarity = ACTIVE / VERIFIED Confirm that a factual query uses the second memory.
Then run a personalization query where the first memory has an allowed personalization capability and show that the result may differ.
Exercise 6 — Assertion event versus proposition Input:
My manager is Alice. Store separately:
verified observation of what the user said unverified world proposition about the manager Write tests proving they receive different capabilities.
The deeper lesson A stateless hallucination is visible at the moment it occurs.
A persistent hallucination can become invisible precisely because it survived.
The database begins to confer legitimacy on text simply by containing it.
That is the trap.
stored ≠ verified active ≠ true retrieved ≠ admissible relevant ≠ authorized repeated ≠ corroborated root ≠ independent source summarized ≠ independently supported response permitted ≠ memory admitted A memory system must preserve the distinctions the rest of this book worked to build.
Its canonical record therefore looks more like:
MEMORY RECORD │ ├── CONTENT │ ├── ORIGIN │ ├── VERIFICATION │ ├── TEMPORAL VALIDITY │ ├── LIFECYCLE │ ├── CLAIM-LEVEL LINEAGE │ ├── SOURCE FAMILIES │ └── CAPABILITIES and the two central policy functions are:
$$ \boxed{ Admission(m,context) \rightarrow \text{persistence capabilities} } $$and:
$$ \boxed{ Use(m,purpose,caller,context) \rightarrow \text{admissible / inadmissible} } $$while:
$$ \boxed{ Retrieved(m) \not\Rightarrow Admissible(m) } $$and:
$$ \boxed{ Persisted(m) \not\Rightarrow Verified(m) } $$and most importantly:
$$ \boxed{ GeneratedByModel(m) + Stored(m) + Retrieved(m) \not\Rightarrow Evidence(m) } $$ Model output must never become evidence merely because the system remembered it.
The final chapter can now assemble the whole architecture.
The question is no longer:
How do we stop a language model from ever hallucinating? That is the wrong systems goal.
The question is:
How do we build a system that assumes its model can fail, detects the failure where possible, limits what survives, and keeps unverified output from becoming action or state?
Related but distinct: training-time feedback Model-generated data entering future training corpora creates a related but mechanically different feedback loop.
Memory contamination changes agent behavior through inference-time persistent context.
Recursive synthetic-data training can change model weights.
The first can often be mitigated by memory invalidation and revocation within one system.
The second can be distributed across datasets and future model versions and may be much harder to reverse.
Shumailov et al. showed degradation under some recursive model-generated training regimes.[8]
Kazdan et al. later showed that collapse is not inevitable under every data workflow and that the outcome depends strongly on how real and synthetic data are retained and accumulated.[9]
The mechanisms differ.
The shared systems pattern is only:
model output → persistent data → future model input → altered future behavior That family resemblance is enough for this chapter.
Research roots Zehao Lin, Xixuan Hao, Renyu Fu, Shaobo Cui, Kai Chen, Chunyu Li, Zhiyu Li and Feiyu Xiong, “A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecycle,” arXiv:2604.16548, 2026. Frames writable persistent memory around lifecycle phases including write, store, retrieve, execute, propagate, and rollback, and emphasizes provenance, versioning, and governance across the lifecycle. https://arxiv.org/abs/2604.16548
Guilin Zhang, Wei Jiang, Xiejiashan Wang, Aisha Behr, Kai Zhao, Jeffrey Friedman, Xu Chu and Amine Anoun, “Adaptive Memory Admission Control for LLM Agents,” ICLR 2026. Treats memory admission as an explicit structured decision rather than automatic retention and evaluates an interpretable admission framework on long-term conversational memory. https://arxiv.org/abs/2603.04549
Xue Tan, Hao Luan, Mingyu Luo, Xiaoyan Sun, Ping Chen and Jun Dai, “RevPRAG: Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysis,” Findings of EMNLP 2025, pp. 12999–13011. Studies poisoning of RAG knowledge databases and detection of poisoned responses. https://aclanthology.org/2025.findings-emnlp.698/
Saksham Sahai Srivastava and Haoyu He, “MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval,” arXiv:2512.16962, 2025. Studies persistent poisoning of experience memory, where malicious successful-looking trajectories are retrieved and imitated on later tasks. https://arxiv.org/abs/2512.16962
Sidharth Pulipaka, Stanislau Hlebik, Leonidas Raghav, Sahar Abdelnabi, Vyas Raina, Ivaxi Sheth and Mario Fritz, “Hidden in Memory: Sleeper Memory Poisoning in LLM Agents,” arXiv:2605.15338, 2026. Studies delayed attacks in which adversarial external context induces fabricated persistent memories that later steer stateful assistants. https://arxiv.org/abs/2605.15338
Haoyu Wang, Guangyuan Dong, He Liang, Zijing Zhang, Jiachen Luo, Chuang Liu, Chao Xue and Hao Tang, “MemGuard: Persisting Verifier Signals for LLM-Agent Memory Governance,” arXiv:2608.21867, 2026. Treats verifier output as persistent lifecycle metadata used in memory admission, retrieval, conflict handling, summarization and archival. https://arxiv.org/abs/2608.21867
“Cognitive Autonomous Memory Security (CAMS) against injection and extraction attacks in long-term memory of AI agents,” Egyptian Informatics Journal 34, 100983, 2026. Proposes a multi-layer memory-security architecture including write guards, provenance, temporal monitoring, graph analysis and periodic re-scanning. https://doi.org/10.1016/j.eij.2026.100983
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal, “AI models collapse when trained on recursively generated data,” Nature 631, 755–759, 2024. Demonstrates degradation under recursive training on model-generated data. https://doi.org/10.1038/s41586-024-07566-y
Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho and Sanmi Koyejo, “Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World,” ICML 2025. Shows that recursive synthetic-data behavior depends strongly on the training workflow and on how real and synthetic data are retained and mixed. https://proceedings.mlr.press/v267/kazdan25a.html
pgvector Project, “pgvector: Open-source vector similarity search for Postgres,” current documentation. Demonstrates vector nearest-neighbor search combined with ordinary SQL metadata filtering and documents the interaction between filtering and approximate vector indexes. https://github.com/pgvector/pgvector
Microsoft Research, “GraphRAG,” current project documentation. Provides a research implementation that extracts entities, relationships and claims, creates graph/community structures and combines those outputs with vector embeddings for retrieval-augmented generation. The project repository describes the implementation as research-oriented and largely in maintenance mode in 2026. https://github.com/microsoft/graphrag
European Union, Regulation (EU) 2016/679, General Data Protection Regulation, Article 17, “Right to erasure (‘right to be forgotten’).” Defines circumstances in which a data subject may obtain erasure of personal data, subject to the regulation’s conditions and exceptions. https://eur-lex.europa.eu/eli/reg/2016/679/oj
California Department of Justice, “California Consumer Privacy Act (CCPA),” current guidance, updated August 28, 2026. Lists consumer rights including the right to delete personal information collected from them, subject to exceptions. https://oag.ca.gov/privacy/ccpa
Richard T. Snodgrass, Developing Time-Oriented Database Applications in SQL, Morgan Kaufmann, 2000; and ISO/IEC 9075:2011 (SQL:2011), which adds application-time-period tables (valid time) and system-versioned tables (system / transaction time). The valid-time vs transaction-time distinction is the standard bitemporal model. https://www2.cs.arizona.edu/~rts/tdbbook.pdf
Next: Building Systems That Distrust Their Models The book now has almost every component it needs.
GENERATION ↓ MEASUREMENT ↓ ADVERSARIAL EVALUATION ↓ MULTI-AXIS DIAGNOSIS ↓ ANSWERABILITY ↓ POLICY ↓ VERIFICATION / REPAIR ↓ MEMORY ADMISSION ↓ ACCESS + ADMISSIBILITY ↓ PERSISTENT STATE The final chapter must assemble these pieces into one engineering architecture.
It must answer:
Which components are deterministic? Which are probabilistic? Where are the trust boundaries? What is logged? What is replayable? What happens when sensors disagree? What happens when verification is unavailable? What may be committed to persistent state? How does the system recover after failure? The final question is:
What does an AI system look like when distrust of its own model is a design principle rather than an emergency patch?