What We Can Now Prove

The final application runs. Its source references resolve, its token budget holds and a real model returns text. Yet none of the final answers satisfies the frozen output contract. Calling the workflow complete would conceal the distinction the book began by drawing.

A working path through software is not proof of a useful result.

Improve and Writer: A Real Retrieval Workflow composed the local application while preserving human control. We can now reconstruct the entire system and ask what its accumulated evidence permits us to claim.

Reconstruct the system we actually executed

The repository contains a progressively developed retriever, not a new production platform. Its final text-question application follows this path:

retained sources + trusted current view + original question
                         |
         canonical identities and whole retrieval units
                         |
           existing lexical / exact dense search
                         |
           fixed hybrid or bounded adaptive loop
                         |
          whole-unit selection and eligible packet
                         |
          versioned contract and current-view recheck
                         |
          actual-tokenizer admission and chat input
                         |
             fixed local consumer generation
                         |
      raw output -> format/citation checks -> answer matching

This schematic matches the executed downstream comparison. Earlier modules also implement controlled segmentation, approximate candidate search, cross-encoder reranking, diversity selection, query transformations, hierarchy and declared relationship traversal. Their earlier experiments remain separately executable. The final text route does not silently incorporate or re-evaluate every mechanism.

Each component answers a failure already encountered. Identity prevents a matching sentence from acquiring the wrong source. Indexing avoids repeated text preparation. Dense retrieval offers a route through vocabulary mismatch. Fusion and reranking operate on candidates that reached them. Selection manages a finite packet. Given updated trusted state, current-view checks can refuse evidence that has become ineligible; the ordinary application does not discover live permission changes. The consumer contract exposes the final gap between selection, actual input and application output.

Keep the negative evidence in the ledger

The earlier independent review results are reported in the author-supplied adjudications. Those are attributed review evidence, distinct from this implementer’s own tests and arithmetic checks. The supplied final-batch adjudication reports independent verification, including four token-for-token consumer regenerations. That is attributed review evidence; author readiness remains a separate decision recorded in the handoff.

Established literature supplies background mechanisms, not measurements of this system. Manual sparse and multi-vector teaching controls are not trained-model comparisons. A proposed improvement remains a hypothesis until an appropriate experiment executes it.

What changed our beliefs

Vocabulary mismatch justified adding another representation. It did not establish that dense search always wins. The retrieval unit also changed the problem: retaining a useful source span did not guarantee that an encoder’s limited input represented it.

On the small external SciFact comparison, hybrid fusion matched lexical cutoff recall but lost early ordering. The cross-encoder changed rankings without repairing the reported aggregate deficit. One query supplied that deficit, and lexical already placed a positive first on most positive questions. These limits narrow the negative finding without erasing it.

The content-budget comparison was more severe. Most relevant whole documents could not fit the chosen allowance. Two selectors consequently returned identical packets with little opportunity to exercise diversity. Equality there is a feasibility limitation, not evidence that diversity has no value.

Complex retrieval demonstrated source-backed traversal of supplied relationships. It did not discover a new graph from arbitrary text. Positive source coverage could coexist with a rejected conflicting path. The constructed final world mirrored development task families, limiting independence.

The bounded controller genuinely observed results and changed its actions. It used 47 calls instead of always continuing’s 68, while mean nDCG at the declared cutoff fell from about .387 to .277. Dense and lexical fixed baselines also exceeded adaptive’s mean quality with fewer calls. Early stops returned three units against a four-result metric, but an equal-prefix analysis still found twelve improvements from continuing on twenty-one stops. Eighteen stopped packets already contained a known positive. Finding one relevant item had been mistaken for enough evidence.

The downstream experiment exposed another limitation. All 140 task-condition generations, including the 84 external final outputs, violated the required format. None contains a citation marker even when case and placement are relaxed, and sixty-six terminate with an end token while invalid. Prompt-specific model non-compliance is established; parser strictness or truncation cannot explain the universal failure. The separate contributions of prompt design, evidence representation and model capability remain unisolated. Semantic correctness under another contract remains unmeasured.

The resulting zero format-constrained scores do not establish semantic equivalence, answer improvement, citation support, causal evidence use or human helpfulness. Development had already produced twenty-eight invalid outputs before final generation, but the plan lacked a predeclared compliance/capability stop gate. That design weakness does not justify retrospectively repairing the completed evaluation. In constructed controls the model also echoed hostile text, although the application granted it no action tools. The final architecture does not cancel these failures.

Return to the evidence chain

Available information is the retained collection within a declared scope. A source can exist without being current or accessible. Missing annotated support in a restricted corpus is not necessarily proof of answer absence.

Retrieved evidence is what a legal action returned. Candidate ceilings show what a later stage might recover, not what it will select. Another scorer cannot recover an absent candidate without another discovery action.

Eligible evidence survives the current source, version and permission checks. This is tested within trusted local views. Live institutional identity and authorisation remain outside the implementation.

Selected evidence survives the retrieval packet’s count and content limits. Its origins and exact references remain inspectable. Selection does not establish compatibility with the consumer’s tokenizer or remaining input capacity.

Admitted evidence appears in the actual constructed input. We now retain that input and its token IDs, rather than assuming the selected list reached the model. The experiment demonstrates losses between those two stages.

Used evidence requires care. We can observe output text and, when valid, cited labels. Neither inclusion nor a citation proves causal reliance. We have not isolated the model’s internal use of a source.

Helpful evidence improves a defined outcome. String agreement is one operational measure; independently assessed correctness and human usefulness require more. The final study does not establish those stronger outcomes.

The arrows remain engineering obligations. Success at one does not imply success at the next.

Run and inspect, rather than trust the summary

Routine checks and a retained-output demonstration are available locally:

python -B -m unittest discover -s tests -v
python -B experiments/part6-system/run.py --verify experiments/part6-system/final
python -B examples/run_retrieval_demo.py --case fact --output .artifacts/fact-inspection.json

The demonstration reconstructs admission and shows retained actual model output. The missing and revoked cases expose no-evidence and changed-permission paths. It does not pretend that reading a saved output generates another answer. Separate documented commands explicitly perform new local inference.

The evidence register and experiment manifests identify sources, model files, configuration, commands, failures and exact outputs. Retained-vector replay reproduces retrieval and tokenised admission; it is not fresh encoding or generation. The recorded environment and cache paths remain material dependencies, and portability to a new machine has not been established.

Capstone: Retrieval as a Pi Extension

The final practical system exposes the book’s retrieval mechanisms to Pi through three native tools. Python owns the algorithms and evidence contract. A small TypeScript extension validates tool parameters, sends one JSON request to Python and returns bounded evidence. It runs no server, indexes no home directory and grants source text no authority to add capabilities.

    flowchart TD
  A[Explicit Pi retrieval tool call] --> B[TypeScript schema and process limits]
  B --> C[Python facade and existing algorithms]
  D[Named authorised local sources] --> C
  C --> E[Current identity and eligibility checks]
  E --> F[Bounded source-qualified tool result]
  F --> G[Optional configured model consumption]
  G --> H[Separate answer and support evaluation]
  

Start from the repository root. The first command builds a lexical representation of exactly the thirty numbered Retrieval manuscripts. The second prepares the original fictional teaching collection with the already pinned local MiniLM model:

python -B -m examples.capstone build --collection book
python -B -m examples.capstone build --collection teaching --dense

The named collections are trusted operator choices. There is no arbitrary root parameter, recursive crawl or model-callable indexing tool. Cache writes stay under .artifacts/capstone/runtime/; source files remain read-only. Versions fingerprint the declared LF-normalised UTF-8 representation, with codepoint spans and parsing identity. Changed source text or eligibility makes the old snapshot unavailable until an explicit rebuild. Missing model dependencies disable dense, hybrid and adaptive search honestly; lexical search needs no model.

Prerequisites differ by step. The book collection, lexical search and milestones 1 and 4 need only Python 3.12 or later, because the facade uses Path.is_junction. Dense, hybrid and adaptive search, and milestones 2, 3 and 5, need the pinned sentence-transformers/all-MiniLM-L6-v2 revision c9745ed1d9f207416be6d2e6f8de32d1f16199bf already cached locally with the file hashes recorded in experiments/part2-semantic/model.json; nothing is downloaded. That record points to the author’s cache directory, so another machine sets RETRIEVAL_MODEL_PATH to its own copy of that snapshot directory, as integrations/pi/README.md describes. Without the model, those capabilities return dense_unavailable. The explicit Pi command, invoke.mjs and milestone 6 also need Node and Pi 1.2.0, the version tested here.

Load the extension explicitly rather than installing it globally:

pi --no-extensions --no-mcp --no-skills --no-prompt-templates --no-context-files -e ./integrations/pi/index.ts --tools retrieval_search,retrieval_open,retrieval_explain

Each Pi tool call starts a Python process, and dense execution may initialise the encoder. One locally captured hybrid round trip took about 14–15 seconds. The Node bridge has a 30-second hard timeout, separate from the Python controller’s cooperative 25-second deadline; a correct setup on slower hardware can still encounter a deadline or timeout. These individual observations are not latency distributions or guarantees.

With your chosen consumer configured, ask Pi to search the book collection for reciprocal rank fusion and show the evidence. retrieval_search takes a query, named collection and strategy, with bounded result count and text. Its response carries the existing retrieval-contract-v1 inside a small versioned transport: original query identity, collection snapshot, complete source/version/unit references, exact spans, method origins, limits and a recent execution ID. Source text is labelled data. Scores do not certify its truth.

Real output is less tidy than the teaching run below. A lexical search of the book collection for that query returned, in order on the manuscript as captured: the PowerShell block in this chapter that contains the query text, a Chapter 17 paragraph on multi-query fusion, the opening paragraph of Chapter 12 and one of its reference entries. Each opened exactly through retrieval_open. The Chapter 12 section that defines the formula was not among the four. The collection’s own command text repeats the query terms, and short block-sized units reward concentrated term overlap, which may contribute; we have not tested that explanation. retrieval_explain shows the one lexical stage and six backend rows behind the list. Every chapter edit changes the snapshot, so rebuild the book index before searching and expect the ordering to differ if the text has changed.

There is also a provider-free way to execute the installed Pi loader and its actual tool adapter:

'{"tool":"retrieval_search","arguments":{"collection":"teaching","query":"reciprocal rank fusion","strategy":"lexical"}}' | node integrations/pi/invoke.mjs
python -B -m examples.capstone.milestones 6 --summary

The captured teaching run returned status: ok; its first evidence document was fusion from capstone:handbook. The milestone then passed that exact reference and snapshot to the real retrieval_open tool, which returned the same source text, and passed the execution ID to retrieval_explain. The explanation recorded one lexical action and its actual origin ranks, selection, budgets and stop reason. A forged source returned evidence_unavailable. The missing-procedure and restricted-sentinel lexical controls returned empty evidence. These are executed observations, not a natural-language agent transcript or evidence that every empty search means absence.

The four-result book list above is a historical snapshot observation. Its sealed references must not be reused as current references after editing the chapters; rebuild and inspect the new snapshot separately. The teaching summaries likewise report their own actual runs rather than promising a fixed ranking.

Opening a result is a separate operation. retrieval_open rechecks exact identity and current eligibility; it never substitutes a similar sentence for an invalid reference. Optional parent expansion returns a verified containing section only if the whole span fits. retrieval_explain reports observable algorithms, not hidden model reasoning. Recent traces are bounded and may expire; a changed snapshot cannot reuse them. Invalid, stale and ineligible evidence share a public-safe refusal rather than revealing a protected inventory.

The sixth milestone completes a trail through the book:

The summary prints a compact account of actual identities, counts, decisions and limitations. Complete JSON evidence is still saved under .artifacts/capstone/runtime/milestones/; omitting --summary retains the existing one-line machine-readable status. In the declared seven-task comparison, hybrid and adaptive search still return candidates for the missing-evidence controls; an additional representation does not supply an answerability decision. The simple positive-document labels are same-author teaching data. Their retrieval coverage is not an independent benchmark or complete evidence sufficiency. No quality victory is needed for the interface to work.

Only mechanisms actually wired into the facade count as integrated. Parent expansion is present; automatic relation extraction, typed graph-path validation, generated rewriting, learned sparse retrieval and cross-encoder reranking remain separate book examples. MMR runs in its milestone, not as another Pi strategy. The adaptive option retains Chapter 21’s predetermined unchanged-query continuation and uncalibrated proxy, not a general autonomous policy.

The tested runtime was Pi 1.2.0. Its actual CLI loaded the extension through an isolated offline session, and its installed loader/tool wrapper executed all three operations. No agent-model turn or paid provider call was made. That missing step remains explicit: a tool result reaching Pi does not prove that a model used it correctly. The extension registers no context-injection hooks or system-prompt mutation. Its source scope also does not sandbox other Pi tools or the operating system.

The one-shot bridge makes reuse inspectable but has costs. Batch demonstration timings that reuse an encoder cannot stand in for interactive latency. Whole evidence text, complete JSON bytes and model tokens are different budgets. The interface enforces codepoint and wire-byte limits, not an unmeasured guarantee about Pi’s eventual prompt tokens. Oversized results and cancelled or failed processes return explicit errors.

This capstone establishes a working retrieval and evidence-delivery path. It does not repair the original 140 malformed TinyLlama outputs, establish semantic answer improvement, resolve the provenance-model review questions or integrate Writer and Improve. Those earlier results and open obligations remain intact.

Assess readiness against properties, not labels

The implementation provides tested local algorithms, traceable source identities, reproducible captures and a reviewable application boundary. Those are useful accomplishments with bounded evidence.

Horizontal scaling, concurrent load, production tail latency, resident-memory limits, live policy discovery, institutional authentication and commercial reliability are not measured. Model portability and dependable consumer answer correctness are not established. Writer and Improve remain not integrated. The Python dense reference timings do not describe optimised search systems generally.

Before institutional operation, the system would need independently reviewed access control and source rights, deployment-specific capacity and failure tests, validated consumer behaviour, representative fresh evaluation and accountable human action gates. None follows automatically from a passing test suite or a diagram connecting the components.

The whole-book developmental, technical and scientific review remains a separate next stage after author first-draft approval. Draft completion does not authorise publication or a rewrite of earlier evidence.

The practical contribution is a way to reason precisely: define the need, establish the available and permitted evidence, compare an appropriate mechanism with a simpler baseline, preserve identity through selection and admission, inspect the output and reject an improvement the evidence does not support.

Retrieval can make evidence discoverable, attributable and inspectable. It does not make an AI application truthful by itself. Whether the application uses that evidence correctly remains a separate obligation, and an honest system keeps the boundary visible even when the last experiment fails.

References and further reading

Background and extensions for this chapter:

• Sandve et al. (2013). Ten Simple Rules for Reproducible Computational Research. PLOS Computational Biology. Supports the final hand-off of inputs, code, versions and retained outputs.
• Thakur et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks. Motivates keeping broad generalisation separate from results on one collection.
• Salemi and Zamani (2024). Evaluating Retrieval Quality in Retrieval-Augmented Generation. arXiv preprint, v1. Extends the remaining question of how retrieval measurements relate to downstream task outcomes.

Related chapters

• Evidence and Verification. Carries the final evidence chain into verification of the user’s goal.