Chapter 13 of 16

Verification, Repair, and Rejection

By Ernan Hughes30 August 2026AI, Hallucination

Chapter 12 gave the system a control plane.

It can now say:

commitment = HOLD
next_action = VERIFY

or:

commitment = HOLD
next_action = RETRIEVE

or:

commitment = HOLD
next_action = REFINE

or:

commitment = DENY
response_mode = REJECTION

Those are policy decisions.

They are not yet recovery implementations.

The obvious implementation is dangerously tempting:

The answer failed a check.
Ask the model to fix it.

But the model that produced the first unsupported statement can produce a second unsupported statement while “correcting” it.

A fluent repair is not evidence that the repair is correct.

A more confident repair is not evidence that the repair is correct.

A longer explanation of the repair is not evidence that the repair is correct.

The system needs a stricter loop:

    graph TD
    CS[candidate state] --> V[verification]
    V --> FR[localized failure record]
    FR --> RC[repair contract]
    RC --> RA[repair action]
    RA --> NCS[NEW CANDIDATE STATE]
    NCS --> IM[invalidate affected measurements]
    IM --> RM[re-measure]
    RM --> RAUTH[re-authorize]

Repair changes the candidate state, so the system must invalidate affected measurements and require authorization again.

The central invariant is:

c_i \xrightarrow{\text{repair}} c_{i+1}

does **not** imply:

\operatorname{authorized}(c_{i+1}).

Instead:

\operatorname{commitment}(c_{i+1}) = \text{UNDECIDED}

until the new state has passed through measurement and policy again.

A repaired claim is still a claim.

Where we are

The book now has:

GENERATOR
    ↓
CANDIDATE
    ↓
MEASUREMENTS
    ↓
RELIABILITY RECORD
    ↓
POLICY
    ↓
COMMITMENT + NEXT ACTION

Chapter 13 implements several of those next actions:

VERIFY
RETRIEVE
REFINE
REJECT_CANDIDATE
ABSTAIN
REVIEW

The engineering problem is not merely:

How do we improve a bad answer?

It is:

How do we improve a bad answer without silently discarding the reliability guarantees we just built?

That requires six disciplines:

1. verify with evidence-bearing signals
2. localize the failure at the right granularity
3. choose a repair whose dependency boundary is understood
4. treat the result as a new candidate state
5. revalidate every affected dependency
6. stop when recovery no longer makes structural progress

1. Start with the failure, not the repair

Consider:

CANDIDATE cand_v1

c1  Q2 revenue was $43.8 million.
c2  Second-half demand is expected to improve.
c3  Q3 revenue was approximately $46 million.

The admissible evidence contains:

Q2 revenue = $43.8 million
second-half outlook = improve

but no Q3 revenue value.

The correct verifier state is not:

c3 = FALSE

because the evidence does not establish that $46 million is false.

It is:

c3 = INSUFFICIENT_EVIDENCE

A deterministic reference controller used in this chapter produced:

cand_v1 {
  'c1': 'SUPPORTED',
  'c2': 'SUPPORTED',
  'c3': 'INSUFFICIENT_EVIDENCE'
}

Two recovery strategies then behaved very differently.

Omission

cand_v1 {
  'c1': 'SUPPORTED',
  'c2': 'SUPPORTED',
  'c3': 'INSUFFICIENT_EVIDENCE'
}
cand_v2 {
  'c1': 'SUPPORTED',
  'c2': 'SUPPORTED'
}
terminal=PERMIT

Hedging

cand_v1 {
  'c1': 'SUPPORTED',
  'c2': 'SUPPORTED',
  'c3': 'INSUFFICIENT_EVIDENCE'
}
cand_v2 {
  'c1': 'SUPPORTED',
  'c2': 'SUPPORTED',
  'c3': 'INSUFFICIENT_EVIDENCE'
}
cand_v3 {
  'c1': 'SUPPORTED',
  'c2': 'SUPPORTED',
  'c3': 'INSUFFICIENT_EVIDENCE'
}
terminal=ABSTAIN reason=CYCLE_DETECTED

The second strategy changed the wording:

Q3 revenue was approximately $46 million.

into:

Q3 revenue may have been approximately $46 million.

The evidence state did not improve.

That is the first demonstration of the chapter’s main rule:

A successful edit is not a successful verification.

2. Verification is not self-confidence

Suppose we ask the generator:

Are you sure?

It may answer:

Yes. Based on the trend, approximately $46 million is reasonable.

Nothing has been verified.

We have sampled another output from the same generative process.

The useful distinction is:

SELF-ASSERTED CONFIDENCE
What does the model say about its own answer?

VERIFICATION
What external or mechanically grounded evidence
bears on the answer?

Strong verification signals can include:

authoritative retrieved evidence
database lookup
schema validation
compiler result
unit test
calculator
formal solver
runtime state
source provenance
human review

A second LLM judge may also contribute.

But it is not automatically independent merely because it has a different role name.

Useful questions include:

generator-independent?
model-family-independent?
source-lineage-independent?
mechanically checkable?

A 2024 critical survey of LLM self-correction found little general evidence for successful prompted intrinsic self-correction outside unusually favorable tasks, while reliable external feedback substantially improves the conditions under which correction works.[1]

So:

Do not confuse another generation with an external or mechanically grounded check.

3. Use a verification state model

Recovery becomes much safer when the verifier preserves why a claim did not pass.

A useful vocabulary is:

State	Meaning
`SUPPORTED`	admissible evidence licenses the claim
`REFUTED`	admissible evidence contradicts the claim
`INSUFFICIENT_EVIDENCE`	current evidence cannot establish the claim
`CONFLICTING`	credible evidence disagrees
`UNRESOLVED`	verifier cannot determine the relation
`NOT_VERIFIABLE`	no available verification procedure can establish the claim

This matters because the recovery action changes.

REFUTED
→ correct or remove the proposition

INSUFFICIENT_EVIDENCE
→ retrieve, narrow to supported content, omit, or abstain

CONFLICTING
→ reconcile / verify sources

NOT_VERIFIABLE
→ review / qualify / abstain according to policy

The Q3 case should therefore produce something like:

verification_record = {
    "candidate_id": "cand_v1",
    "claim_id": "c3",
    "claim": "Q3 revenue was approximately $46 million.",
    "state": "INSUFFICIENT_EVIDENCE",
    "evidence_ids": ["filing_q1", "filing_q2"],
    "missing_evidence": ["q3_revenue_value"],
    "verification_method": "required_field_check",
    "verifier_version": "financial-verifier-v4",
}

That is far more useful than:

BAD

because it constrains what a legitimate repair is allowed to do.

The recovery flow based on verification state is:

    graph TD
    V[Verification State] --> REF[REFUTED] --> COR[correct or remove the proposition]
    V --> INS[INSUFFICIENT_EVIDENCE] --> RET[retrieve, narrow, omit, or abstain]
    V --> CON[CONFLICTING] --> REC[reconcile / verify sources]
    V --> UNV[NOT_VERIFIABLE] --> REV[review / qualify / abstain]

Repair is keyed to verification state because different epistemic failures require different recovery operations, not a single generic rewrite.

4. Verify the smallest unit that contains the failure — but no smaller

Claim-level verification is often ideal.

But not every failure is claim-local.

Consider:

c1: Drug X was associated with lower symptom scores.
c2: The study was observational.
c3: Therefore Drug X should be prescribed.

Each sentence can be individually grammatical and factual while the argument remains unjustified.

Or:

all individual claims = SUPPORTED
response-level omission = decisive adverse evidence absent

So the verification unit may be:

claim
claim–evidence edge
claim pair
argument
response
trajectory

The correct rule is:

Verify the smallest unit that contains the failure—but no smaller.

The same principle determines whether a local repair is appropriate.

5. Choose the repair surface from demonstrated failure locality

Local repair is not inherently safer.

It is safer when:

failure is localized
dependencies are known
protected content remains valid

Then:

prefer local repair

But if the failure comes from:

false global premise
dense downstream dependence
broken argument structure
response-level omission

then a local sentence patch can preserve the real defect.

A practical decision rule is:

    graph TD
    LF[LOCAL FAILURE] --> KD[KNOWN DEPENDENCIES] --> LP[local patch / omission / rebind]
    GF[GLOBAL PREMISE FAILURE] --> LR[larger-region rebuild / full regeneration / review]
    UF[UNKNOWN DEPENDENCY BOUNDARY] --> LR

Repair scope should therefore be selected from the verified failure record, not from a general preference for small edits.

6. Measure both surface and dependency blast radius

The original repair-blast-radius idea remains useful:

B_{\text{surface}} = \frac{\text{changed answer units}} {\text{total answer units}}.

The unit might be:

claims
sentences
structured fields
code hunks

But textual change is not the whole risk.

Changing one token:

did
→
did not

may invalidate:

polarity
support
summary conclusion
recommendation
policy decision

So define a second quantity:

B_{\text{dep}} = \frac{\text{dependent checks invalidated}} {\text{relevant checks}}.

A repair can therefore have:

low surface blast radius
high dependency blast radius

or the reverse.

For reliability, the second often matters more.

7. Repair operations are different types

A recovery controller should distinguish at least:

Repair type	What changes
Evidence repair	evidence snapshot
Candidate patch	localized proposition/text
Decomposition repair	claim granularity
Attribution repair	source binding
Structural repair	role, polarity, quantity, time
Scope repair	strength or domain of assertion
Omission	unsupported content removed
Full regeneration	whole candidate rebuilt

REFINE is a policy route (Chapter 12). Repair is the operation that route executes. One route can map to several repair types.

The policy route provides a useful starting map:

RETRIEVE
→ evidence repair

VERIFY
→ resolve evidence / source / relation state

REFINE
→ candidate / structure / scope repair

REJECT_CANDIDATE
→ discard version

ABSTAIN
→ stop asserting

But the recovery operator still needs a contract describing exactly what it may alter.

8. Retrieval changes the evidence snapshot, not truth

Suppose policy returns:

HOLD + RETRIEVE

The system retrieves a new document.

That does not repair the candidate automatically.

The new material must itself pass:

source admissibility
freshness
provenance
relevance
conflict checks

Then create:

evidence_snapshot_v2

and invalidate every measurement that depended on evidence_snapshot_v1.

That can include:

support checks
provenance checks
conflict checks
freshness checks
containment geometry
downstream policy decision

The safe sequence is:

    graph LR
    CV[candidate_v1] --> EG[evidence gap] --> RET[retrieve]
    RET --> VE[validate evidence] --> ES[evidence_snapshot_v2]
    ES --> IM[invalidate dependent measurements] --> RV[re-verify candidate_v1]

New evidence cannot simply be appended to the old verdict; it creates a new evidence snapshot that forces dependent measurements to be checked again.

New evidence can even make previously accepted claims less secure by revealing:

contradiction
newer information
source conflict
scope limitation

Corrective RAG is relevant here because it explicitly evaluates retrieval quality and triggers corrective retrieval behavior rather than treating initial retrieval as automatically adequate.[3]

9. Omission is a repair only if the remaining response stays sound

Sometimes the strongest repair is deletion.

If no evidence establishes Q3 revenue:

Q3 revenue may have been around $46 million.

is not a repair.

It is the same unsupported proposition under weaker rhetoric.

Qualification does not convert an unsupported fact into a supported fact.

But deletion can create another class of failure.

Consider:

Revenue grew.
However, margins shrank.
This was due to inflation.      [REMOVE]
Therefore, we must cut costs.

Deleting the third sentence can leave:

Therefore

without a valid premise.

Likewise, removing a sentence may create:

dangling pronoun
broken causal link
missing caveat
misleading partial presentation

So omission requires a cohesion and salvage-integrity pass.

A partial answer is valid only if the retained content remains:

grammatically coherent
referentially closed
logically coherent
not materially misleading through omission

This may require a surface-realization edit after factual deletion.

That edit creates a new candidate state too.

10. Partial salvage needs dependency closure

Suppose the candidate contains claims:

C=\{c_1,c_2,\ldots,c_n\}.

Let each claim have application-defined utility:

u(c_i)\ge0.

We can conceptualize partial salvage as:

S^* = \arg\max_{S\subseteq C} \sum_{c_i\in S}u(c_i)

subject to:

every retained critical claim is authorized
response-level obligations remain satisfied
cohesion constraints hold

But claims may depend on each other.

Let:

c_j\rightarrow c_i

mean that retaining $c_i$ requires retaining, substituting, or explicitly accounting for $c_j$.

Then the retained set must satisfy dependency closure.

This prevents a superficially factual partial answer from becoming misleading because a required premise, adverse caveat, or qualification was removed.

The objective is therefore not:

retain the most sentences

but:

maximize verified retained utility subject to authorization and coherence.

11. Repair contracts should contain executable invariants

A repair contract can be typed as:

repair_contract = {
    "target_claim_ids": ["c3"],
    "allowed_operations": [
        "OMIT_CLAIM",
        "REPLACE_WITH_EVIDENCE",
        "NARROW_TO_SUPPORTED_SCOPE",
    ],
    "protected_claim_ids": ["c1", "c2"],
    "invariants": [
        "company_id(child) == company_id(parent)",
        "reporting_period(child) == reporting_period(parent)",
        "currency(child) == currency(parent)",
        "all_protected_claims_remain_supported(child)",
    ],
    "must_not_add": [
        "new_unverified_financial_values",
    ],
    "max_surface_blast_radius": 0.33,
}

The phrase:

narrow scope

needs precision.

This is legitimate:

Q3 revenue was $46 million.
        ↓
The supplied evidence establishes Q2 revenue at $43.8 million,
but does not establish Q3 revenue.

This is not:

Q3 revenue may have been $46 million.

The first reduces the assertion to what the evidence licenses.

The second merely hedges an unsupported proposition.

12. Every repair creates a new candidate state

This is the chapter’s canonical wording.

A deletion may shrink the assertion surface.

A citation correction may not introduce a new factual proposition.

A deterministic patch may change only one structured field.

Yet all create a new state:

cand_v1
→
cand_v2

and trust attaches to the versioned state, not to the lineage as a whole.

So:

\operatorname{authorization}(c_i) \not\Rightarrow \operatorname{authorization}(c_{i+1})

even if the difference is one token.

A repair record should preserve provenance:

repair_record = {
    "candidate_id": "cand_v2",
    "parent_candidate_id": "cand_v1",
    "parent_verification_record": "verify_1842",
    "repair_action": "OMIT_CLAIM",
    "changed_claim_ids": ["c3"],
    "evidence_snapshot_uri": "cas://evidence/7f...",
    "evidence_snapshot_hash": "sha256:...",
    "repair_model": "generator-v17",
    "repair_prompt_version": "repair-v5",
}

The hash proves integrity.

The immutable snapshot or content-addressed URI makes replay possible.

13. Dependency-aware remeasurement is an invalidation problem

A repair controller should think like an incremental build system.

Suppose:

dependencies = {
    "c3": [
        "polarity_check",
        "support_check",
        "summary_conclusion",
        "response_cohesion",
        "policy_decision",
    ],
}

Editing c3 invalidates those checks.

A cache key should conceptually include:

subject/candidate-component hash
evidence snapshot hash
detector version
detector configuration
calibration version where relevant

For example:

measurement_key = (
    claim_hash,
    evidence_hash,
    verifier_version,
    verifier_config,
)

A cached measurement may be reused only when all of its dependencies are unchanged.

Policy is always recomputed because the reliability record itself may have changed.

14. Repair-induced regression is the Whac-A-Mole failure

A repair can fix one problem and create another.

ORIGINAL
Company A acquired Company B in March 2024.

Suppose only the date is wrong.

A careless patch produces:

Company B acquired Company A in April 2024.

The date has improved.

The relation has regressed.

This is the familiar Whac-A-Mole pattern of generative repair.

Define the set of previously passing properties on candidate $c_i$ as:

\mathcal P(c_i).

At candidate level, regression severity can be measured as:

RSI(c_i,c_{i+1}) = \frac{ |\mathcal P(c_i)\setminus\mathcal P(c_{i+1})| }{ \max(1,|\mathcal P(c_i)|) }.

Across a repair benchmark $\mathcal D$, define a case-level repair-induced regression rate:

RIR_{\text{case}} = \frac1{|\mathcal D|} \sum_{d\in\mathcal D} \mathbb I \left( |\mathcal P(c_i)\setminus\mathcal P(c_{i+1})|>0 \right).

And a property-level rate:

RIR_{\text{property}} = \frac{ \text{previously passing exposed checks that become failures} }{ \text{previously passing exposed checks} }.

Report these on **valid repair attempts**.

A malformed patch should not be mistaken for evidence about the same failure mechanism.

And do not use generic embedding similarity as the preservation oracle.

A relation reversal can remain semantically similar.

Protected-content checks should preserve:

claim identity
roles
polarity
quantities
time
scope
declared invariants

15. External feedback and critics are themselves fallible sensors

The correction loop has two hard tasks:

1. identify what is wrong
2. produce a better candidate

The first is often the bottleneck.

The TACL self-correction survey emphasizes the importance of reliable feedback.[1]

CRITIC demonstrates a tool-interactive pattern in which external tools provide feedback used for revision rather than relying only on introspection.[2]

RePAIR starts from a related problem: agentic RAG systems can rely on critic categories or correction plans that are themselves wrong, and it instead learns response-to-action mappings without requiring an explicit fine-grained taxonomy.[5]

The lesson is not:

typed failure states are useless.

It is:

A component does not become an oracle because we call it a critic.

Evaluate critics with:

failure-localization precision
failure-localization recall
false repair-trigger rate
correct recovery-action rate

Self-RAG provides another useful contrast: retrieval, generation, and learned reflection signals can be integrated into one trained model.[6]

That can improve behavior.

It does not remove the architectural question:

Which signal is allowed to authorize the repaired candidate?

Reflection remains evidence for a decision, not authorization by itself.

16. Full regeneration needs a taint boundary

Sometimes the entire argument is built on a false premise.

Then local repair is inappropriate.

Use:

repair_mode = FULL_REGENERATION
blast_radius = WHOLE_CANDIDATE

But do not feed the failed answer back as unquestioned factual context. Give it a typed record, the same shape Chapter 14 uses for memory, so the boundary travels with it:

tainted_context = {
    "candidate_id": "cand_v3",
    "origin": "FAILED_CANDIDATE",
    "verification_state": "REFUTED",       # or INSUFFICIENT_EVIDENCE
    "allowed_uses": [
        "FAILURE_LOCALIZATION",
        "USER_INTENT_RECONSTRUCTION",
        "STYLE_CONSTRAINT",
        "PROTECTED_STRUCTURE",
    ],
    "forbidden_uses": [
        "FACTUAL_EVIDENCE",
        "SOURCE_AUTHORITY",
        "PROPOSITION_SEED",
    ],
    "parent_verification_record": "verify_1842",
}

origin = FAILED_CANDIDATE is not an authoritative external source and not an authorized derived artifact — the lineage classes from Section 21. The regeneration step reads allowed_uses; a step that tries to consume the failed candidate as FACTUAL_EVIDENCE fails the same check a memory read would (Chapter 14).

A safer regeneration request is:

Generate a new answer from the verified evidence package.
The previous candidate is available only as a record of failures
and user-intent constraints.
Do not treat its factual assertions as evidence.

After regeneration:

new candidate
→ full affected measurement bundle
→ policy

No inherited trust.

17. A bounded recovery controller needs budgets, progress, and cycle detection

Attempt count is only one resource.

A real recovery budget may include:

max_repair_attempts
max_retrieval_queries
max_verifier_calls
max_critic_tokens
max_wall_clock_latency
max_monetary_cost
max_human_escalations

The controller should also detect repeated states.

For example, store:

candidate_hash
verification_state_vector
repair_action

and stop if the same candidate/failure state reappears.

Repeated state is evidence that the current recovery strategy has stopped learning anything.

The reference controller in this chapter detects exactly that.

The hedge repair repeatedly changes rhetoric without changing evidence status, producing:

terminal=ABSTAIN reason=CYCLE_DETECTED

Progress should not be one weighted “quality” score.

Use structural progress such as:

hard blockers decreased?
critical unresolved claims decreased?
repair regressions stayed zero?
evidence completeness increased?
policy state moved toward PERMIT?
verified retained utility increased?

One conceptual progress vector is:

\Phi(c) = ( N_{\text{hard blockers}}, N_{\text{critical unresolved}}, N_{\text{regressions}}, -\text{verified utility} ).

The exact ordering is application-specific.

The principle is not:

keep trying while score rises

but:

continue only while a declared reliability dimension is genuinely improving.

The recovery loop with budgets and cycle detection:

    graph TD
    START[start] --> VER[verify candidate]
    VER --> ALL{all claims SUPPORTED?}
    ALL -- yes --> PERMIT[PERMIT]
    ALL -- no --> SIG[compute signature]
    SIG --> SEEN{signature seen?}
    SEEN -- yes --> ABSTAIN[ABSTAIN: CYCLE_DETECTED]
    SEEN -- no --> BUDGET{repair budget left?}
    BUDGET -- no --> ABSTAIN2[ABSTAIN: BUDGET_EXHAUSTED]
    BUDGET -- yes --> REPAIR[apply repair operation]
    REPAIR --> VER

The loop permits repair only while it can produce a new verification state; repeated signatures and exhausted budgets terminate in abstention instead of endless revision.

18. A small deterministic recovery controller

Reference environment

Environment: Python 3.13
Dependencies: Python standard library only
Canonical implementation: experiments/hallucination/recovery_demo.py

Run it with:

python experiments/hallucination/recovery_demo.py

The core controller is intentionally small:

from dataclasses import dataclass
from enum import Enum
import hashlib

class V(str, Enum):
    SUPPORTED = "SUPPORTED"
    INSUFFICIENT_EVIDENCE = "INSUFFICIENT_EVIDENCE"

@dataclass(frozen=True)
class Claim:
    id: str
    text: str
    requirement: str

@dataclass
class Candidate:
    id: str
    parent: str | None
    claims: list[Claim]
    authorized: bool = False

def verify(candidate, evidence):
    return {
        c.id: (
            V.SUPPORTED
            if c.requirement in evidence
            else V.INSUFFICIENT_EVIDENCE
        )
        for c in candidate.claims
    }

def signature(candidate, states):
    body = "|".join(f"{c.id}:{c.text}" for c in candidate.claims)
    h = hashlib.sha256(body.encode()).hexdigest()[:12]
    return h, tuple(sorted((k, v.value) for k, v in states.items()))

The loop then, in outline:

while True:
    states = verify(current, evidence)

    if all(v == V.SUPPORTED for v in states.values()):
        current.authorized = True
        return "PERMIT", current

    sig = signature(current, states)

    if sig in seen:
        return "ABSTAIN:CYCLE_DETECTED", current
    if repairs >= max_repairs:
        return "ABSTAIN:REPAIR_BUDGET_EXHAUSTED", current
    if verifier_calls >= max_verifier_calls:
        return "REVIEW:CRITIC_BUDGET_EXHAUSTED", current

    seen.add(sig)
    current = REPAIRS[mode](current, states)   # a new candidate; no inherited authorization

Two repair modes exercise the loop.

omission removes every claim the verifier could not support. The omission trace is:

cand_v1 {'c1': 'SUPPORTED', 'c2': 'SUPPORTED', 'c3': 'INSUFFICIENT_EVIDENCE'}
cand_v2 {'c1': 'SUPPORTED', 'c2': 'SUPPORTED'}
terminal=PERMIT

hedge softens the wording of the unsupported claim without adding evidence. The second hedge is idempotent, so the candidate and its verification state repeat, and cycle detection fires:

cand_v1 {'c1': 'SUPPORTED', 'c2': 'SUPPORTED', 'c3': 'INSUFFICIENT_EVIDENCE'}
cand_v2 {'c1': 'SUPPORTED', 'c2': 'SUPPORTED', 'c3': 'INSUFFICIENT_EVIDENCE'}
cand_v3 {'c1': 'SUPPORTED', 'c2': 'SUPPORTED', 'c3': 'INSUFFICIENT_EVIDENCE'}
terminal=ABSTAIN reason=CYCLE_DETECTED

The controller also rejects an undeclared repair mode before any repair runs:

--- mode=guess ---
ValueError: unknown repair mode: 'guess'

This demonstration proves controller invariants under a perfect structured oracle.

It does not establish the accuracy of a real-world verifier or repair model.

Those require separate evaluation.

19. Human review should receive the failed recovery path, not only the final text

When policy routes to:

REVIEW

the human reviewer should not receive only:

cand_v3

They need a review package:

original candidate
current candidate
diff
target failed claim
verification state
evidence used
repair attempts
new failures introduced
budget reason
policy reason

For example:

review_package = {
    "candidate": "cand_v3",
    "parent_chain": ["cand_v1", "cand_v2"],
    "target_failure": "INSUFFICIENT_Q3_REVENUE",
    "attempts": [
        "HEDGE_CLAIM",
        "HEDGE_CLAIM",
    ],
    "terminal_reason": "CYCLE_DETECTED",
    "evidence_snapshot": "cas://evidence/7f...",
}

The human is then resolving a diagnosed reliability problem rather than reconstructing the entire history manually.

20. The recovery benchmark should decompose the bottleneck

A minimum benchmark should report:

Dimension	Metrics
Verification	localization precision/recall, false repair triggers
Repair	targeted fix rate, RIR, new unsupported claims
Preservation	protected invariants, dependency regressions, cohesion
Efficiency	surface/dependency blast radius, steps, budget exhaustion
Routing	correct recovery operation, unnecessary regeneration/review
System	final authorized coverage, latency, retrieval/verifier cost

Add two critical baselines.

No-repair baseline

What happens if the system:

omits
abstains
or rejects

without running a generative repair stage?

Oracle-repair baseline

What would happen with:

perfect localization
perfect gold repair

These decompose:

verification bottleneck
vs
repair-generation bottleneck.

And preserve Chapter 7’s held-out discipline:

known repair regression suite
held-out same-mechanism suite
novel-failure recovery set

Once a repair case enters the regression suite, repeatedly fixing that exact case is not evidence of general recovery ability.

21. Repair candidates are ephemeral until authorized

This is the bridge to the next chapter and should already be an invariant here.

A candidate under repair must not be written into a persistent evidence store merely because it exists.

cand_v1 HOLD
cand_v2 HOLD
cand_v3 HOLD

are ephemeral working states.

Only an explicitly authorized artifact may cross the persistence boundary.

Otherwise the recovery system can manufacture its own future evidence:

unsupported candidate
→ persisted
→ indexed
→ retrieved during repair
→ appears independent
→ "verifies" descendant candidate

That is circular contamination.

Evidence snapshots therefore need lineage that distinguishes:

authoritative external source
authorized derived artifact
unverified model output

and repair candidates must never silently cross between those classes.

22. Using AI: ask for a bounded patch, then distrust the patch

If an LLM acts as the repair operator, constrain its task.

A weak instruction is:

Improve this answer and make it more accurate.

A stronger contract is:

The verifier identified exactly one failed claim: c3.

Verification state:
INSUFFICIENT_EVIDENCE

Available evidence:
- c1 SUPPORTED by source_1
- c2 SUPPORTED by source_2
- no admissible Q3 revenue value exists

Allowed operations:
- omit c3
- replace c3 only if a supplied source directly supports replacement

Protected claims:
- c1
- c2

Forbidden:
- new financial values
- changes to company identity
- changes to reporting period

Return:
1. repaired answer
2. changed claim IDs
3. operation used

A characteristic bad repair would be:

Q3 revenue may have been approximately $45.8 million.

That may sound cautious.

The verifier should still return:

INSUFFICIENT_EVIDENCE

The surrounding system owns the truth conditions.

After the patch:

Run the verifier again rather than trusting the repair model’s declaration that it succeeded.

23. What you should now be able to answer

After this chapter, you should be able to explain:

Why every repair creates a new candidate state even when the assertion surface shrinks.
The difference between REFUTED and INSUFFICIENT_EVIDENCE.
Why external or mechanically grounded feedback is different from another model completion.
Why repair granularity should match the smallest verified failure boundary.
The difference between surface blast radius and dependency blast radius.
Why omission requires cohesion and salvage-integrity checks.
Why new retrieval invalidates all evidence-dependent measurements.
Why a repaired candidate cannot inherit authorization.
How repair-induced regression and cycle detection expose failed recovery loops.
Why repair candidates must remain ephemeral until explicitly authorized.

Worked question — Why is deletion often safer than hedging?

Suppose the evidence does not contain Q3 revenue.

The original claim is:

Q3 revenue was $46 million.

Changing it to:

Q3 revenue may have been $46 million.

changes the rhetoric but does not add evidence for the underlying Q3 value.

If the verification contract requires an admissible source for reported revenue, both claims remain:

INSUFFICIENT_EVIDENCE.

Deletion removes the unsupported assertion.

But deletion is authorized only after the remaining response passes cohesion, dependency, and policy checks.

So the correct rule is not:

delete whenever uncertain.

It is:

remove unsupported material
when doing so preserves the obligations of the remaining response.

This is the deletion side of conformal factuality (Chapter 6 §9). Mohri and Hashimoto derive a guarantee by backing a generation off — removing or generalizing claims — until the surviving text clears a factuality bound.[7] Hedging a claim does not shrink the entailment set it commits to; deleting it does.

24. Exercises

Exercise 1 — Local versus global repair

Create a five-claim answer with one localized unsupported claim.

Compare:

local omission
local evidence-backed patch
whole-answer regeneration

Report:

surface blast radius
dependency blast radius
target repair success
new regressions

Exercise 2 — Distinguish evidence states

Create examples for:

REFUTED
INSUFFICIENT_EVIDENCE
CONFLICTING
UNRESOLVED

Specify the correct recovery route for each.

Exercise 3 — Repair-induced regression

Start with:

Company A acquired Company B in March 2024.

where only the date is wrong.

Repair the date and verify that:

subject
object
polarity
quantity

remain unchanged.

Compute candidate-level RSI.

Exercise 4 — Dependency invalidation

Create a dependency graph for:

claim
support
summary
recommendation
policy

Edit one claim and identify which cached measurements remain valid.

Exercise 5 — Cycle detection

Implement two repair operators that oscillate between two candidate states.

Confirm that:

(candidate_hash, verification_state)

cycle detection terminates the loop before the general attempt budget.

Exercise 6 — Partial salvage integrity

Create an answer where deleting one unsupported claim leaves a dangling:

therefore
however
this
it

Repair the cohesion without adding new factual assertions.

Re-run verification afterward.

25. The deeper lesson

The naive repair loop is:

model makes mistake
→ tell model it made a mistake
→ model says something else
→ trust the new answer

That is not a reliability architecture.

A reliable recovery loop is:

    graph TD
    CS[candidate state] --> EB[evidence-bearing verification]
    EB --> LD[localized diagnosis]
    LD --> RC[repair contract]
    RC --> BR[bounded repair]
    BR --> NCS[new candidate state]
    NCS --> DI[dependency invalidation]
    DI --> RM[re-measurement]
    RM --> REG[regression check]
    REG --> POL[policy]

The system continues only while recovery makes real structural progress.

If it does not, the correct output may be:

omit
review
abstain
reject

That is not the system giving up.

It is the system refusing to turn repair effort into unjustified commitment.

A repaired claim is still a claim.

It still needs evidence.

It still needs structural fidelity.

It still needs provenance.

It still needs epistemic adequacy.

It still needs policy authorization.

And until it receives that authorization, it remains transient working state—not tomorrow’s evidence.

That final distinction leads directly to the next problem.

Research roots

Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han and Rui Zhang, “When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs,” Transactions of the Association for Computational Linguistics 12, 1417–1440, 2024. Finds weak general evidence for intrinsic prompted self-correction, identifies feedback generation as a major bottleneck, and reports stronger correction behavior when reliable external feedback is available. https://aclanthology.org/2024.tacl-1.78/
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan and Weizhu Chen, “CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing,” ICLR 2024. Uses external tools to evaluate aspects of generated outputs and feeds tool results back into iterative revision. https://www.microsoft.com/en-us/research/publication/critic-large-language-models-can-self-correct-with-tool-interactive-critiquing/
Shi-Qi Yan, Jia-Chen Gu, Yun Zhu and Zhen-Hua Ling, “Corrective Retrieval Augmented Generation,” arXiv:2401.15884, 2024. Adds retrieval evaluation and corrective retrieval, including web search and decompose-then-recompose processing, rather than assuming the initial retrieved corpus is sufficient. https://arxiv.org/abs/2401.15884
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan and Kelvin Guu, “RARR: Researching and Revising What Language Models Say, Using Language Models,” ACL 2023. Retrieves attribution for generated text and post-edits unsupported content while explicitly trying to preserve unaffected material. https://aclanthology.org/2023.acl-long.910/
Gongbo Zhang, Yifan Peng and Chunhua Weng, “Improving Retrieval-Augmented Generation without Taxonomy-based Error Categorization,” ACL 2026, pp. 155–165. Introduces RePAIR, motivated partly by unreliable critic categorization and ineffective correction, and learns response-to-action plans for flawed RAG outputs. https://aclanthology.org/2026.acl-short.14/
Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil and Hannaneh Hajishirzi, “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection,” ICLR 2024. Integrates adaptive retrieval, generation and learned reflection signals into one model, providing a useful contrast to externally separated verification architectures. https://proceedings.iclr.cc/paper_files/paper/2024/hash/25f7be9694d7b32d5cc670927b8091e1-Abstract-Conference.html
Christopher Mohri and Tatsunori Hashimoto, “Language Models with Conformal Factuality Guarantees,” ICML 2024. Derives a factuality guarantee by backing a generation off — removing or generalizing claims — until the surviving text clears a conformal bound. https://proceedings.mlr.press/v235/mohri24a.html

Next: The Memory Contamination Problem

Chapter 13 keeps untrusted repair states inside a bounded loop.

But production systems also write state.

They create:

conversation memory
user profiles
agent scratchpads
vector-store documents
summaries
knowledge-graph facts
database rows
tickets
reports
future prompts
training examples

Now imagine an unverified candidate crossing that persistence boundary.

On the next run, the system may retrieve its own output as evidence.

Then the failure changes form:

hallucination
→ persistent assertion
→ retrieved context
→ apparent evidence
→ reinforced hallucination

The next chapter asks:

How do we stop an unverified output from becoming tomorrow’s evidence?