Part VI — Debugging Prompts, Retrieval, and Hallucinations

The ledger chunk exists — and the answer says it does not

Chapters 30–31 versioned and minimized the prompt. Now the minimal prompt still fails: asked about refund RB-8814, the assistant invents a “processed” status while the ledger line “refund PENDING, no reference issued” sits in the corpus. The prompt is exonerated. The words arrived; the evidence did not.

Concrete failure. The retriever returns five chunks for “refund RB-8814 status.” The ledger chunk is not among them. The engineer re-embeds everything, the failure persists, and the team debates the embedding model — while the actual break sits one stage downstream, invisible because “retrieval” is treated as one box.

OBSERVATION: corpus contains the ledger line (byte-verified by direct lookup); top-5 retrieval log for the pinned query lacks it; assembled context therefore lacks it; output invents it. HYPOTHESIS H1 (chunking): the ledger line was split or truncated so no chunk holds it whole. H2 (index/staleness): the chunk exists but the queried snapshot predates it. H3 (ranking): the chunk is indexed but scores below top-k. INFERENCE: none yet — H1/H2/H3 predict different per-stage probe outcomes and are separable only by opening the pipeline stage by stage.

This chapter’s question: which retrieval stage first diverged — chunk, embed, index, or rank — and what contract did it violate?

The debugging object is now Evidence

Chapters 30–31 debugged the instruction. From here the object changes. Of the book’s five debugging objects — values, state, distributions, evidence, trajectories (top.txt) — Part VI’s second half works the fourth. The earlier questions were what value is wrong?, what state produced it?, what distribution of behavior appears? Now the questions are about evidence:

What evidence existed? Where did it come from? Was it preserved through ingestion and parsing? Was it selected by the retriever? Was it included in the assembled context? Was it attributed to the right source? Did it actually support the claim — and why was that claim accepted?

Naming evidence as the object does not make evidence true. Retrieved bytes can be stale, mutually contradictory, incomplete, attributed to the wrong document, or simply insufficient for the conclusion drawn from them. The debugging task keeps its shape — it is Chapter 2’s checkpoint-the-chain method (probe each boundary in execution order, stop at the first mismatch), now run over a retrieval pipeline instead of a data pipeline — but the path is an evidence chain, not a call stack.

Across Chapters 32–35 the records these methods produce compose into one accumulating artifact, the evidence ledger: for a given claim or decision, its source and source version, the retrieval event that surfaced it, whether it entered the rendered context, how it was attributed, the support relation to the final claim, and the acceptance decision. Not every row carries every field. The ledger can establish whether evidence existed at a boundary, whether it was preserved, whether a claim connects to supplied support, and where an evidence-chain transition first failed. It cannot establish universal truth, a causal mechanism inside the model, correctness because a source was retrieved, or sufficiency because a citation exists. It is Part VI’s addition to a case that is still accumulating — the reproducible execution bundle (Part IV), the AI work-product case file (Part V), and now the claim-to-evidence lineage.

Why “fix the embeddings” fails first

The obvious move — swapping or re-training embeddings — fails because retrieval is four machines in series, and a blanket fix repairs the wrong one. Four defects hide behind single-box thinking:

1. Chunk invisibility. The ledger line is split across two chunks (“refund PEND-” / “-ING, no reference…”), so no embedding of either half matches the query. Re-embedding split text re-buries the same shards. Chunk granularity is a studied design axis, not a detail: Chen and colleagues found that indexing a corpus by fine-grained propositions — atomic, self-contained factoids — retrieves materially better than passage-level chunks and improves downstream QA (Chen et al., 2024). A split ledger line is a granularity failure with a known class of fix.
2. Snapshot confusion. The chunk was indexed Tuesday; the query ran against Monday’s snapshot. Fresh embeddings indexed into a stale serving snapshot change nothing observable.
3. Rank cutoff. The ledger chunk ranks 47th of 50; top-k=5 discards it. The retriever “worked” (it found the chunk) and the cutoff discarded the work. A better embedder shuffles ranks without moving the cutoff.
4. Assembly conflation. Retrieval returned the chunk but context assembly truncated it (Chapter 33’s boundary B). Blaming retrieval repairs a stage that held up its contract.

OPINION: “the retriever failed” is not a diagnosis. It is a confession that you have not looked inside. Name the stage or name nothing.

The mental model: retrieval-as-pipeline — and the full pipeline is longer than most diagrams admit:

source → ingest → parse → chunk → embed → index → query construction → retrieve → rerank / select → compile context → generate → verify

Evidence can be lost or corrupted at any boundary: an ingestion job that skips a document, a parser that drops a table out of a PDF, a query builder that never includes the reference number. This is a diagnostic decomposition, not a required architecture — collapse the stages your stack does not have. This chapter’s lab holds ingestion, parsing, and indexing fixed and begins at query time: chunk → embed → index → rank. That is an experimental boundary, not a claim that upstream stages cannot fail. Part VI’s rule is find the first evidence-chain divergence, and a document missing from the index is an evidence failure even when retrieval behaves perfectly over the defective index — which is why the first probe below byte-verifies that the needed line is actually in the corpus before any stage is examined.

The first-divergence rule applied to evidence flow: the earliest stage whose output violates its contract is the break; downstream stages are suspects only after it holds.

The method: per-stage contracts and probes

Freeze the query, corpus snapshot, and parameters. Then walk forward:

1. Chunk contract. Every source line needed by a fixture must survive chunking whole (or with sufficient overlap) in at least one chunk. Probe: direct lookup — grep the chunk store for the ledger string; if no chunk contains it intact, H1 convicted here, stop.
2. Embed contract. The chunk’s stored vector must equal a fresh embedding of its bytes under the pinned model (detects model swaps and corruption). Probe: re-embed the chunk bytes, compare vectors exactly.
3. Index contract. The chunk must be searchable in the snapshot the query actually hit. Probe: fetch by ID from the serving snapshot (not the build snapshot); version IDs must match. Mismatch → H2 convicted.
4. Rank contract. Given the query embedding, the chunk must score within the served top-k — or the cutoff must be justified. Probe: score the query against the full snapshot, report the ledger chunk’s exact rank and the score gap to rank k. Rank 47 of 50 with top-k=5 → H3 convicted; repair is cutoff, filtering, or query shaping — not re-embedding.

    flowchart TD
    V["byte-verify the needed source line is in the corpus; freeze query + snapshot + top-k"] --> C{"chunk store holds the line intact in one chunk?"}
    C -->|no| H1["H1 chunking — split / truncated; fix chunk sizes + overlap, stop"]
    C -->|yes| E{"stored vector == fresh embedding of the chunk bytes?"}
    E -->|no| ES["embed contract broken — model swap / corruption"]
    E -->|yes| I{"fetch by ID from the SERVING snapshot: found, version matches?"}
    I -->|"missing / version skew"| H2["H2 index / staleness — serve what was built, stop"]
    I -->|yes| RK{"chunk ranks within the served top-k?"}
    RK -->|"no, rank > k"| H3["H3 ranking / cutoff — query shaping / metadata filter / cutoff, NOT re-embedding"]
    RK -->|yes| ASM["all contracts hold — route to context assembly / generation attribution (Ch33)"]
  

RAG STAGE PROBES (query frozen; snapshot idx-2026-08-14; top-k=5):
chunk store grep "refund PENDING, no reference": 1 chunk HIT (hash d4e2, intact) -> H1 EXONERATED
re-embed d4e2 vs stored vector: IDENTICAL -> embed contract HOLDS
fetch d4e2 from SERVING snapshot: FOUND (version match) -> H2 EXONERATED
full-snapshot rank of d4e2: 47/50; score gap to rank 5: 0.31 (cosine, as MEASUREMENT only)
RULE: rank is located here, not diagnosed here. Scores locate; contracts convict.

OBSERVATION (constructed illustration, not a measured run): chunk intact, vector identical, snapshot matched, rank 47/50 against top-k=5. UPDATED BELIEF: H3 supported for this instance (ranking/cutoff stage diverged first); H1/H2 exonerated here; embed-swap repair rejected. No universal retriever claim.

Score values above are MEASUREMENTs for locating the chunk, never a diagnosis: a similarity number does not explain why the rank is low, and raising top-k without a fixture check trades precision for recall blind.

Example: tracing the ledger chunk through four stages

The refund query is frozen; the corpus snapshot pinned; analysis is mechanical:

# RAG pipeline inspector sketch: stage probes in order (no repair yet)
chunk = chunk_store.grep("refund PENDING, no reference")  # OBSERVATION: hit d4e2 or UNKNOWN
assert chunk.bytes_intact  # H1 probe: whole line in one chunk?
fresh = embed(chunk.bytes, model=pinned_model)  # MEASUREMENT: fresh vector
assert fresh == chunk.stored_vector  # embed contract: exact match required
served = serving_snapshot.fetch("d4e2")  # H2 probe: present in HIT snapshot?
assert served.version == chunk.version  # snapshot identity, not recency impression
ranking = serving_snapshot.rank(query_emb, top_n=50)  # H3 probe: full ranking
log(ranking.position("d4e2"), ranking.gap_to_k(5))  # MEASUREMENT: 47, gap 0.31
# Predictions pre-written: H1 predicts grep MISS; H2 predicts fetch MISS/version
# skew; H3 predicts rank > k with contracts 1-3 holding. Only one can be first.

In the constructed case all probes pass until rank: the ledger chunk is intact, correctly embedded, and indexed — but buried at 47 because the query terms (“RB-8814 status”) match ticket chatter better than ledger phrasing (“refund PENDING”). The targeted repairs are stage-correct: add the ledger’s reference vocabulary to the query-shaping rules, or add a metadata filter (source=ledger) for refund-status fixtures — then re-run the Chapter 30 suite. Re-embedding the corpus would have churned every vector and fixed nothing.

Second artifact: the rank-gap decision record

Rank probes return a number (position 47, gap 0.31) and numbers invite threshold superstition (“anything below 0.5 is broken”). The decision record converts the measurement into a stage-correct choice without letting the score become the diagnosis:

1. Record position, gap, and cutoff together. d4e2: rank 47/50, gap-to-k5 0.31 (cosine), k=5. The gap is a MEASUREMENT for locating effort, never a cause. Two incidents with identical gaps can route differently (one needs a filter, one needs query shaping).
2. List the stage-correct options with predictions. Cutoff raise (predicts: recall up, precision unmeasured — must re-run suite); metadata filter (predicts: ledger fixtures recover, ticket fixtures unaffected); query shaping (predicts: this phrasing family recovers). One option per intervention series.
3. Run the cheapest discriminating option first. Filters before re-embedding, shaping before model swaps. Cost order is retrieval-specific repair hygiene: cutoff/filter/shaping changes touch configuration, embedder changes touch every vector.
4. Re-probe, don’t re-assume. After the repair, re-rank the frozen query and confirm the chunk’s new position by measurement; “should rank higher now” is a FORECAST awaiting its OBSERVATION.

RANK DECISION (constructed illustration, query frozen):
options: (a) k=5->20 predicts d4e2 served; (b) source=ledger filter predicts rank 1-3;
(c) re-embed predicts unknown (rejected: contracts 1-3 hold, no evidence for embed fault)
chosen: (b) filter -> re-probe: d4e2 rank 2/50, suite 12/12 x3
RULE: the cheapest stage-correct repair runs first. The most expensive theory waits longest.

OBSERVATION (constructed illustration): the filter repair recovers the refund fixtures 3/3 while ticket-chatter fixtures stay green; the deferred re-embed is later shown unnecessary when the suite holds for a month. UPDATED BELIEF: filter supported for this query family and snapshot; no claim about optimal-k in general. One family’s recovery is not a tuning law.

Research lineage: the stages are real failure axes, and “more relevant” is not always the fix

The rank-47 case is the vocabulary problem. Furnas and colleagues showed decades ago that two people spontaneously choose the same word for the same thing less than a fifth of the time (Furnas et al., 1987). A query phrased “RB-8814 status” and a ledger line phrased “refund PENDING, no reference issued” are two people naming the same fact differently. Query shaping, hybrid lexical+dense retrieval, and hypothetical-document expansion (HyDE) are all responses to this specific problem — and none of them is “re-embed the corpus.”

Context composition matters, and adding documents is not free. Cuconasu and colleagues found that the set of documents placed in the prompt strongly shapes RAG output, to the point that adding random documents improved accuracy on their setup — and that near-miss “distracting” documents hurt more than random ones (Cuconasu et al., 2024). The random-document result did not robustly replicate: a 2026 re-run found it highly sensitive to prompt formulation, output-length limits, and model generation, with much of the reported gain tracing to truncation and malformed outputs rather than a real effect of noise (Mazuryk et al., 2026). The distractor half is the robust and operative one, and its lesson for the rank-gap decision record stands: raising top-k to recover one buried chunk also admits several near-miss distractors, so the cutoff-raise option must be re-run against the full suite, not waved through because recall went up.

The stages map to a known failure catalogue. Barnett and colleagues’ seven RAG failure points (Chapter 4) are this pipeline’s stages enumerated: missing content (ingest/parse), top-ranked missed and relevant-chunk-not-retrieved (chunk/embed/rank), not-in-context (assembly — Chapter 33), and the generation-side three (Barnett et al., 2024).

Lab 32: stage-bisection with pre-written probe predictions (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own stage-probe log.

Setup. Take one failing query where the needed source line is byte-verified present in the corpus. Pin the query bytes, corpus snapshot, embedding-model identifier, index version, top-k, and seed. The pipeline stage is the independent variable under test; query, snapshot, and parameters are controlled.

Task.

1. Before probing, write H1/H2/H3 with distinct predicted probe patterns: H1: “chunk-store grep MISSES the intact line”; H2: “grep HITS but serving-snapshot fetch MISSES or version-skews”; H3: “grep HITS, fetch HITS, rank > k with gap ___.”
2. Run the four probes in order, ≥3 trials for the rank probe (embeddings deterministic under pin; ranking checked for query-shaping stability). Record OBSERVATION (hit/miss, hashes, ranks verbatim) and UPDATED BELIEF per hypothesis.
3. Repair only the first-diverged stage; re-run the suite before touching any other stage.

Success criterion. A stage-probe log naming the first-diverged stage with contract verdicts per stage, plus a stage-correct repair gated on the suite. An embedding swap with no probe log is explicitly not completion.

Companion tool: RAG Pipeline Inspector

What it accepts: the frozen query, corpus snapshot ID, chunk store with hashes, embedding-model identifier, serving-snapshot ID, rank configuration (top-k, filters), and the needed-source line for the fixture. What it performs: it runs the four stage probes in order (grep → re-embed compare → serving fetch → full-snapshot rank), records per-stage contract verdicts with hashes and versions, and blocks stage-skipping repairs (no rank repair before chunk/index verdicts). What it can establish: which stage first violated its contract for the examined query and snapshot — and which stages held. What it cannot establish: why the rank is low in semantic terms, corpus completeness in general, or future query reliability. It never treats similarity scores, confidence values, single-run outputs, agreement across retries, or downstream symptoms as diagnosis. How its output changes your next action: H1 routes to chunking repair (sizes, overlap, boundary rules); H2 routes to snapshot/version repair (serve what was built); H3 routes to cutoff/filter/query-shaping repair; all-hold routes to assembly/generation attribution (Chapter 33).

Paper form, sufficient for this chapter:

Query hash: ___  Snapshot: build ___ / serving ___ (match? y/n)
Chunk: HIT/MISS (hash ___) | Embed: IDENTICAL/SKEW | Fetch: FOUND/MISS (ver ___)
Rank: ___/___ vs top-k=___ (gap ___)  FIRST DIVERGENCE: chunk/embed/index/rank
NEXT REPAIR (one stage only): ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Probe in order; repair the first break.

Reusable procedure: bisect every retrieval failure

1. Freeze and byte-verify — query, snapshot, needed line confirmed in corpus.
2. Probe chunk — grep the store; intact chunk or stop here.
3. Probe embed and index — vector identity; serving-snapshot fetch with version match.
4. Probe rank — full-snapshot position and gap to k (measurement, not verdict).
5. Repair the first divergence only — one stage, then the suite.

Failure modes

• Embedder reflex. Re-embedding before chunk/index verdicts. Churn without location.
• Snapshot mirage. Probing the build index while the query hit a stale serving copy. Version IDs decide; recency impressions do not.
• Score-as-cause. “Similarity 0.31, therefore irrelevant.” Scores locate candidates; contracts convict stages.
• Cutoff blindness. Tuning vectors while top-k discards the evidence. The cutoff is a stage too.
• Multi-stage repair. Fixing chunking and ranking together. One stage per test or attribution is lost.
• Assembly misattribution. Repairing retrieval for evidence the retriever returned but assembly dropped. Chapter 33 settles that boundary.
• Filter neglect. Reaching for re-embedding before trying a metadata filter or cutoff change. Configuration repairs cost vectors; vector repairs cost corpora.
• Threshold superstition. Promoting a rank-gap number into a general cutoff law (“below 0.5 is broken”). Gaps locate effort per query; they legislate nothing.

Limits, per contract: one inspection covers one query, one snapshot, one configuration; it does not certify the retriever, does not transfer across queries, and stays UNKNOWN where the snapshot or chunk store lacks versioning.

References

• Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense X Retrieval: What Retrieval Granularity Should We Use? Proceedings of EMNLP, 2024, pp. 15159–15177. https://aclanthology.org/2024.emnlp-main.845/
• George W. Furnas, Thomas K. Landauer, Louis M. Gomez, and Susan T. Dumais. The Vocabulary Problem in Human-System Communication. Communications of the ACM 30(11), 1987, pp. 964–971. https://doi.org/10.1145/32206.32212
• Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The Power of Noise: Redefining Retrieval for RAG Systems. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, pp. 719–729. https://doi.org/10.1145/3626772.3657834
• Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. Seven Failure Points When Engineering a Retrieval Augmented Generation System. Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering (CAIN), 2024, pp. 194–199. https://doi.org/10.1145/3644815.3644945
• Michał Mazuryk, Fleur Dolmans, Louis Gehringer, Ina Klaric, Jia-Huei Ju, and Mohammad Aliannejadi. The Powerless Noise: How Experimental Settings Shape the Reported Power of Noise. arXiv:2607.03615, 2026. https://arxiv.org/abs/2607.03615

Debugging Checklist

• Query, corpus snapshot, model ID, and top-k frozen and logged?
• Needed source line byte-verified in corpus (not remembered)?
• Chunk probe run (intact-chunk HIT/MISS with hash)?
• Embed probe run (fresh vs. stored vector identity)?
• Index probe run against the SERVING snapshot (version match)?
• Rank probe run (exact position + gap to k, ≥3 trials where stochastic)?
• First-diverged stage named; no scores, confidence, single runs, agreement, or symptoms cited as cause?
• Cheapest stage-correct repair tried before vector-level churn?
• Cutoff-raise repairs re-run against the full suite (distractors admitted, not just recall gained)?
• Post-repair rank re-measured (not assumed)?

What This Chapter Established

• Retrieval-as-pipeline with four checkable contracts (chunk intactness, embed identity, index servability, rank within cutoff) and ordered probes — demonstrated on the constructed ledger-chunk case (rank 47 vs. top-k=5), no measured runs claimed.
• The first-divergence application to evidence flow: earliest violated contract is the break; later stages wait.
• Lab 32 as a proposed stage-probe log the reader executes; the RAG Pipeline Inspector contract (accepts/performs/can-establish/cannot-establish/next-action).
• The rank-gap decision record as the repair discipline: cheapest stage-correct option first, re-probed by measurement.
• What was NOT proved: any embedder ranking, any retrieval-quality claim in general, or any fix for other queries. One query bisected; nothing universal.
• Research grounding: chunk granularity is a studied failure axis and propositions beat passages (Chen et al.); the rank-47 case is the classic vocabulary problem — query/document term mismatch (Furnas et al.) — addressed by shaping, hybrid retrieval, or HyDE, not re-embedding; context composition strongly shapes output and near-miss distractors hurt more than random noise (Cuconasu et al. — the separate “random documents help” result did not robustly replicate, Mazuryk et al.), so a cutoff-raise must be re-suite-gated; the stages map to Barnett et al.’s seven RAG failure points.
• Position in the arc: prompts versioned (30) and minimized (31); now the evidence supply line is staged. Stages mapped, contracts posted.

Next

Staged retrieval tells you whether the evidence was found. It does not settle who lost it when the answer is still wrong with retrieval holding — the handoff to context assembly and generation remains. Chapter 33, “Retriever Failure or Generator Failure?,” runs the full three-artifact attribution the book previewed in Chapter 3; which boundary diverged first on the refund triple is its chapter’s to establish, not this one’s.