Part VI — Debugging Prompts, Retrieval, and Hallucinations

“I retrieved the ledger, confirmed RB-8814, and answered” — none of which happened

Chapter 3 stated the rule; Chapters 30–34 built the machinery that makes it enforceable. Now the capstone case. After the invented RB-8814 answer, the engineer asks the model to explain itself. It replies with a crisp chain-of-thought: “First I retrieved ticket #4471 and the refund ledger; both confirmed RB-8814; therefore I answered processed.” Every step is fluent. The retrieval log shows no ledger chunk. Ticket #4471 is another customer’s. The explanation is a second fabrication about the first fabrication — and it arrives shaped exactly like a trace.

Concrete failure. The incident ticket is closed with “Root cause: ledger confirmed refund (per model’s explanation).” Next sprint, the team “fixes” ledger indexing. The ledger was never read. Two fabrications now anchor a repair, because behavior (a generated paragraph) was filed as execution (instrumented records).

OBSERVATION: model-generated explanation asserts ledger retrieval + confirmation of RB-8814; frozen retrieval log lacks the ledger hash; frozen sent context lacks RB-8814 (0 hits); output’s claims already verdict UNSUPPORTED (Chapter 34 table). HYPOTHESIS H1 (faithful trace): explanation steps correspond to instrumented records. H2 (plausible confabulation): explanation is fluent behavior uncorrelated with records. H3 (anchoring contamination): reading the explanation first biases the human verdict toward H1. INFERENCE: none yet — H1/H2/H3 predict different explanation-vs-record alignments and are separable only by auditing each explanation step against hashes.

This chapter’s question: for each step of the model’s explanation, what instrumented record corroborates it — or does the step stand uncorroborated?

Why “ask it to show its work” fails first

The obvious move — requesting step-by-step reasoning and checking whether it sounds right — fails because chain-of-thought is output, not instrumentation. Five defects hide behind work-showing:

1. Trace-shape mimicry. Numbered steps, “first I retrieved… then I compared…,” ticket IDs in monospace — the typography of a log with none of a log’s properties (no timestamps, no hashes, no tool records).
2. Post-hoc plausibility. The explanation is generated after (and conditioned on) the answer. It rationalizes the conclusion; it does not replay the computation. A different answer would have received an equally fluent, different “trace.”
3. Faithfulness overclaim. The chapter makes no quantitative claim about how often explanations are faithful in general. The operational stance is narrower and sufficient: this explanation is untrusted until each step links to a record, whatever population statistics may or may not say. That said, the published measurements are not reassuring: when Anthropic researchers slipped problem-solving hints into prompts and checked whether the models’ chains-of-thought admitted using them, Claude 3.7 Sonnet acknowledged the hint about 25% of the time and DeepSeek R1 about 39% — and outcome-based reinforcement learning raised faithfulness only partway before plateauing (Chen et al., 2025). Those figures are for two reasoning models on one experimental setup; the stance does not depend on them, but they are why it is not paranoia.
4. Agreement theater. “Explain again” returns the same story, and consistency is filed as corroboration. The same failure mode sampled twice is repetition, not proof (Chapter 3’s rule, unchanged).
5. Contamination order. Reading the explanation before the records anchors judgment: the log’s missing ledger hash starts looking like “probably a logging gap” instead of what it is — absence of evidence, honestly marked UNKNOWN unless logging completeness itself is instrumented.

OPINION: chain-of-thought is the model telling you what a good answer process would sound like. Gratitude is appropriate. Citation is not.

The mental model: explanation-as-behavior — generated text about a process, debugged like any other output (claim-split, span-demanded, verdict-per-step), never executed like a trace. The book’s rule, final form for LLM systems: only controlled instrumentation counts as trace evidence; everything else is a hypothesis awaiting hashes.

The method: the explanation-audit protocol

Freeze the explanation plus the Chapter 33 triple (retrieval log, sent context, output) and the Chapter 34 claim table. Then:

1. Quarantine first. Log the explanation hash; mark every “because / therefore / I retrieved” sentence as EXPLANATION-class before opening any record (Chapter 3’s two columns, enforced).
2. Split into checkable steps. One asserted action or fact per row (“retrieved ticket #4471,” “ledger confirms RB-8814,” “therefore processed”).
3. Demand a record per step. Each row must link to an instrumented artifact (log line with timestamp, chunk hash, tool-call record, context offset) or be marked UNCORROBORATED. No link, no trace status.
4. Run the order test. Audit blind where possible: verdict the records before re-reading the explanation (H3 probe — explanation-first vs. records-first notes compared for verdict drift).
5. File behavior, not trace. Corroborated steps graduate to evidence; uncorroborated steps remain behavior-to-debug (prompt, retrieval, or generation hypotheses) — never root cause.

    flowchart TD
    Q["quarantine: hash the explanation, class every 'because / I retrieved' sentence as EXPLANATION"] --> B["blind pass: verdict the records (triple + claim table) BEFORE unsealing the explanation"]
    B --> SP["split the explanation into checkable steps — one asserted action or fact per row"]
    SP --> L{"per step: links to an instrumented record (log line + timestamp, chunk hash, tool call, offset)?"}
    L -->|yes| CO["CORROBORATED — graduates to evidence"]
    L -->|"no link"| UN["UNCORROBORATED — stays behavior-to-debug, never root cause"]
    CO --> OP{"order probe: explanation-first verdict differs from records-first?"}
    UN --> OP
    OP -->|yes| CON["contamination — log it, fix the records-first process"]
    OP -->|no| FILE["file: corroborated steps as evidence, the rest as behavior exhibit"]
  

EXPLANATION AUDIT (explanation hash e771; records frozen per Ch33):
E1 "I retrieved ticket #4471" -> record: retrieval log HAS #4471 (hash 77aa, rank 1) -> CORROBORATED (retrieved, relevance unjudged)
E2 "I retrieved the refund ledger" -> record: ledger hash d4e2 ABSENT from log (rank 47, below k) -> UNCORROBORATED (contradicted as retrieval claim)
E3 "both confirm RB-8814" -> record: 0 hits RB-8814 in log + sent context + snapshot -> UNCORROBORATED (claim already UNSUPPORTED, Ch34)
E4 "therefore processed" -> record: sent context says "PENDING" -> explanation's inference CONTRADICTED by records
RULE: one corroborated step (E1) does not launder three uncorroborated ones. Steps verdict individually.

OBSERVATION (constructed illustration, not a measured run): one of four explanation steps links to a record; two assert retrievals/confirmations with zero supporting bytes; the concluding inference contradicts the frozen context. UPDATED BELIEF: H1 rejected for this explanation (no step-to-record correspondence as a whole); H2 supported (fluent confabulation with one coincidentally true retrieval); H3 testable via the order probe below — verdict drift, if present, confirms contamination susceptibility for this debugger, not a universal rate. No population claim made.

No detail, coherence, confidence, re-explanation agreement, single corroborated step, or downstream calm (“the customer accepted it”) promotes the uncorroborated steps to trace.

Example: auditing the RB-8814 self-report step by step

The explanation is frozen; the audit is mechanical and deliberately uncurious about prose quality:

# explanation-vs-trace audit sketch: steps against records (no re-asking yet)
expl = load("explanation@e771.md")  # OBSERVATION: 4 steps, EXPLANATION-class
records = load_triple()  # retrieval log, sent context, output, all hashed (Ch33)
table = load_claim_table()  # per-claim verdicts (Ch34): C4a/C4b UNSUPPORTED
for step in split_steps(expl):
    links = link_to_records(step, records)  # MEASUREMENT: hashes/offsets or NONE
    verdict = "CORROBORATED" if links else "UNCORROBORATED"
    log(step.text, links or "NONE", verdict)
# H1 predicts: every step links. H2 predicts: >=1 load-bearing step links NONE.
# H3 probe: two auditors (or one auditor twice, order swapped) verdict records
# blind vs explanation-first; drift recorded as OBSERVATION, not averaged away.

In the constructed case the audit leaves exactly one honest sentence (“I retrieved ticket #4471” — true, and also the wrong ticket for this customer, a retrieval-relevance issue for Chapter 32’s methods, not a trace credential). The ticket is rewritten: root cause reverts to UNKNOWN pending the Chapter 33 split test (which in incident C convicted generation), the ledger-indexing repair is cancelled, and the explanation is attached as behavior exhibit E-771, labeled as such.

Second artifact: the records-first team discipline

Individual auditors can hold the quarantine line; teams leak it through process — explanations arrive in tickets, chats, and reviews pre-attached, and every reader absorbs them before the records. The discipline is environmental:

1. Records attach before explanations in every ticket. Retrieval log hash, sent-context hash, output hash, claim table — filed first. Explanations attach as exhibits with EXPLANATION-class labels. A ticket whose first artifact is a model paragraph is returned unprocessed, no matter how urgent.
2. Blind the first verdict. The on-call engineer verdicts the triple and claim table before opening the explanation exhibit. The explanation is unsealed only for the audit step. Order is enforced by template, not willpower.
3. Review the audit, not the story. Design and incident reviews walk the per-step link table (E1 CORROBORATED @hash…, E2 NONE…), never the explanation’s narrative. “It makes sense” is ruled out of order; “it links” is the only admissible praise.
4. Track contamination events honestly. When an explanation-first read demonstrably moved a verdict (H3 probe positive), log it as a process finding: INC-120: explanation-first draft blamed ledger indexing; records-first re-audit convicted generation. The log justifies the template’s friction to the next skeptic.

TICKET TEMPLATE (order enforced):
1. records (hashes): log ___ sent ___ out ___ claim-table ___
2. blind verdict (records only): ___
3. explanation exhibit (hash ___): per-step links E1 ___ E2 ___ E3 ___ E4 ___
4. order probe: drift y/n ___ 5. filed as: evidence steps ___ / behavior ___
RULE: the template decides the order. Urgency never promotes exhibits to evidence.

OBSERVATION (constructed illustration): three sprints under the template produce two logged contamination events, both caught at step 3 before repair; the ledger-indexing misrepair class stops recurring. UPDATED BELIEF: template supported as contamination control for this team-instance; no claim about explanation quality trends. Process held; population unjudged.

Research lineage: introspection is not a trace, for people either

The model is not uniquely broken. Nisbett and Wilson’s classic review found that people have little or no direct introspective access to their own higher cognitive processes: when asked why they did something, subjects construct a plausible causal story from implicit theories about what should have influenced them, and they do this confidently even when the real influence is demonstrably something else (Nisbett & Wilson, 1977). A model’s post-hoc “here is how I reached the answer” is the same kind of artifact — a plausible account, not a replay. The explanation-audit protocol is what interviewers of human witnesses already know: check the account against the record.

Faithfulness has a test methodology, and the audit uses it. Atanasova and colleagues proposed operational tests for whether a natural-language explanation reflects the model’s actual reasoning — counterfactual tests (does changing the input in a way the explanation says is decisive actually change the output?) and reconstruction tests (can the explanation alone reproduce the prediction?) (Atanasova et al., 2023). The chapter’s per-step record linkage is the record-grounded version; the order-swap probe is a counterfactual test aimed at the auditor rather than the model.

This is Chapter 3, updated. Turpin’s biased-CoT result and Lanham’s load-bearing test established the rule for 2023-era models; the Anthropic 2025 result extends it to explicit reasoning models. Nothing in the intervening research licenses trusting a chain-of-thought as a trace. The field’s current position is not that chain-of-thought is worthless but that it is a partial and fragile oversight signal: Korbak and colleagues — a large cross-organization group — argue CoT monitoring can catch some intent to misbehave and is worth investing in precisely because it is imperfect, and warn that scaled reinforcement learning and latent-reasoning architectures could erode even that (Korbak et al., 2025). Being monitorable for intent is not the same as being a faithful replay of the computation. The explanation-audit protocol — link each step to a record, keep only the corroborated subset — is how a practitioner extracts the partial value without over-trusting it.

Lab 35: explanation audit with order-swap contamination probe (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own audit table.

Setup. Take one incident with a wrong answer, frozen triple, claim table, and a model-generated explanation (chain-of-thought, “sources,” or post-hoc justification). Pin everything. The audit target is the explanation step (independent variable across rows); records are controlled. A second factor — audit order — is also controlled by swap.

Task.

1. Before auditing, write H1/H2/H3 with distinct predicted patterns: H1: “every explanation step links to a record hash”; H2: “≥1 load-bearing step links to NONE (0 hits)”; H3: “explanation-first notes contain ≥1 verdict that records-first notes mark UNCORROBORATED/UNKNOWN.”
2. Audit twice: pass 1 records-first (verdict the triple and claim table blind), pass 2 explanation-first (re-verdict with explanation visible). Record OBSERVATION (per-step links, per-pass verdicts verbatim) and UPDATED BELIEF. Repeat the pinned input ≥3 times where generation is involved; explanation stability is reported, never averaged into truth.
3. File the explanation as behavior with its audit table; no repair cites an uncorroborated step.

Success criterion. A per-step audit table with record links or explicit NONEs, plus the order-swap comparison. A “the explanation checked out” paragraph with no table is explicitly not completion.

Companion tool: Explanation-vs-Trace Checker

What it accepts: the frozen model explanation (hash), the frozen triple (retrieval log, sent context, output), the claim table, and the two-pass audit notes. What it performs: it enforces EXPLANATION-vs-EVIDENCE class separation, requires a record link per explanation step (or explicit UNCORROBORATED), runs the order-swap comparison, and blocks any diagnosis that cites an unlinked explanation sentence as a handoff fact. What it can establish: whether each step of the examined explanation is corroborated by instrumented records — and whether audit order moved the verdicts. What it cannot establish: why the model generated the explanation, explanation faithfulness rates in general, or anything about future explanations. It never treats fluency, detail, confidence, agreement across re-explanations, single corroborated steps, scores, or downstream symptoms as trace evidence. How its output changes your next action: corroborated steps join the evidence set for Chapters 32–34 routing; uncorroborated steps route back to prompt/retrieval/generation hypotheses as behavior; order drift routes to process repair (records-first discipline, blind audit) before any technical repair.

Paper form, sufficient for this chapter:

Explanation hash: ___  Records hashes: log ___ / sent ___ / out ___
Per-step links: E1 ___ E2 NONE E3 NONE E4 CONTRADICTED
Order probe: records-first ___ vs explanation-first ___ (drift y/n ___)
FILE AS: evidence steps ___ / behavior exhibit ___  NEXT: ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Links before belief.

Reusable procedure: audit every explanation you rely on

1. Quarantine the explanation — hash it, class it, read records first.
2. Split into steps — one asserted action/fact per row.
3. Link or mark — record hash per step or UNCORROBORATED, individually.
4. Probe the order — records-first vs. explanation-first; record drift honestly.
5. File correctly — corroborated steps are evidence; the rest is behavior exhibit.

Failure modes

• Trace-filing. Pasting “I retrieved X” into the evidence column. Behavior laundered as instrumentation is the chapter’s named failure.
• Single-step laundering. One true step (“retrieved #4471”) credentialing the whole story. Steps verdict alone.
• Re-ask corroboration. “It explained the same way twice.” Repetition is stability of behavior, not correspondence with records.
• Faithfulness-rate import. Citing a vendor claim or remembered study (“models are 90% faithful”) as this-step evidence. Population claims never verdict instances; vendor claims are attributed, not inherited.
• Logging-gap charity. “The ledger read probably wasn’t logged.” Absence is UNKNOWN, and UNKNOWN never corroborates. Fix logging, then re-audit.
• Agent-trace preview error. Treating this chapter’s output text as an agent trajectory verdict. Trajectories (tool calls, state, transitions) are Part VII’s object; what trajectory analysis establishes is Chapter 36’s to establish, not this one’s.
• Exhibit promotion. Letting a well-formatted explanation slide from exhibit to evidence because the ticket is long and the paragraph is short. Format never changes class; only links do.
• Template bypass under urgency. Skipping records-first “just this once” for a severe incident. Severe incidents are where contamination costs most — the template’s friction scales with the stakes it protects.

Limits, per contract: one audit covers one explanation against one frozen record set; it establishes correspondence, not causation or generality; it stays UNKNOWN where logging completeness is itself uninstrumented.

References

• Yanda Chen, Joe Benton, Ansh Radhakrishnan, et al. Reasoning Models Don’t Always Say What They Think. Anthropic, arXiv:2505.05410, 2025. https://arxiv.org/abs/2505.05410
• Richard E. Nisbett and Timothy D. Wilson. Telling More Than We Can Know: Verbal Reports on Mental Processes. Psychological Review 84(3), 1977, pp. 231–259. https://doi.org/10.1037/0033-295X.84.3.231
• Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. Faithfulness Tests for Natural Language Explanations. Proceedings of ACL, 2023, pp. 283–294. https://aclanthology.org/2023.acl-short.25/
• Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS, 2023. https://arxiv.org/abs/2305.04388
• Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, et al. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv:2507.11473, 2025. https://arxiv.org/abs/2507.11473

Debugging Checklist

• Explanation frozen with hash, classed EXPLANATION before records opened?
• Triple + claim table frozen (Ch33/Ch34 artifacts referenced by hash)?
• Every explanation step split and linked to a record or marked UNCORROBORATED?
• H1/H2/H3 predictions pre-written with distinct link patterns?
• Order-swap probe run (records-first vs. explanation-first drift recorded)?
• Stability across ≥3 trials reported (not averaged into truth)?
• Ticket order enforced (records before explanation exhibit)?
• Contamination events logged as process findings when drift is found?
• No fluency, confidence, agreement, scores, single corroborated steps, or symptoms cited as trace?

What This Chapter Established

• The explanation-audit protocol for LLM systems: quarantine, step-split, per-step record linkage, order-swap contamination probe — demonstrated on the constructed RB-8814 self-report (1/4 steps corroborated), no measured runs and no population faithfulness claim.
• The book rule in final Part VI form: chain-of-thought is behavior to debug (Chapters 30–34 route it); only controlled instrumentation is trace.
• Research grounding: measured CoT hint-acknowledgement is ~25–39% for two reasoning models even after RL (Chen et al., 2025); humans confabulate reasons for their own behavior with equal confidence (Nisbett & Wilson) — the model is not uniquely broken; the audit’s step-linkage and order-swap are the record-grounded and counterfactual forms of published faithfulness tests (Atanasova et al.); the field treats CoT as a partial, fragile oversight signal (Korbak et al.), and the audit protocol is the safe-extraction mechanism. This is Chapter 3 (Turpin, Lanham) extended, not revised.
• Lab 35 as a proposed audit record the reader executes; the Explanation-vs-Trace Checker contract (accepts/performs/can-establish/cannot-establish/next-action).
• The records-first team discipline as the contamination control: ordered ticket template, blind first verdict, logged drift events.
• What was NOT proved: any faithfulness rate, any logging-completeness claim, any agent-trajectory result. One explanation audited; nothing universal.
• Position in the arc: Part VI closes here — prompts versioned (30), minimized (31), retrieval staged (32), boundaries attributed (33), claims verdict (34), explanations quarantined (35). Evidence debugged end to end. The evidence ledger is complete: every claim traceable to a source, a retrieval event, a context inclusion, and an attribution — or to the boundary where that chain first broke.

Next

Evidence now has hashes, stages, boundaries, claim verdicts, and quarantined explanations — but every one of those artifacts in this Part described a single prompt-response pass. The systems that fail next do not answer once; they act across steps, call tools, keep state, and drift over time. Chapter 36, “An Agent Is a Trajectory,” proposes the trajectory as the next debugging object; what trajectory evidence shows is its chapter’s to establish, not this one’s.