← Retrieval From First Principles

The Reference Retrieval Contract

Turn source identity, permitted scope, selection and admission limits into explicit, testable promises to a consumer.

The evidence packet still points to the right sentence. Between retrieval and prompt construction, its source becomes restricted. An application that trusts the packet’s earlier eligibility decision can now send unauthorised evidence to its model.

Nothing about the ranking score identifies that failure. The reference still resolves historically, and the sentence has not changed. What changed is permission to admit it now.

Does Better Retrieval Improve the Task? connected selection to an actual consumer. That connection needs a contract: what may the application depend on, what must it recheck, and how should an unresolved obligation be represented?

Make promises before choosing fields

Our contract makes five narrow promises. It preserves the original need, restricts the search to caller-authorised sources, identifies the exact selected evidence, makes resource limits inspectable and refuses to treat an incomplete stage as success at a later stage.

These promises require more than a list of text strings. The application needs the original query and identity, request and execution identities, strategy, trusted view, source allowance, selected references and spans, rank origins, stop reason and limitations. It also needs a separate admission record identifying what entered the consumer input.

The implementation in examples/consumer_retrieval.py names its schema retrieval-contract-v1. This is a local implemented protocol, not a proposed industry standard. A consumer that understands this version can reject another version rather than interpreting unfamiliar fields optimistically.

The response is deliberately smaller than the internal controller trace. Public evidence and safe status belong in the response. Candidate inventories, intermediate diagnostic state and private policy reasons do not become public merely because they are useful for debugging.

The request identity binds the query, trusted view and source allowance. The execution identity also binds the strategy, selected results and stop information. Changing the query after retrieval therefore changes the object the consumer is validating.

These identities are hashes, not signatures. An attacker able to replace the object can recalculate a digest. Validation still needs a trusted caller, a canonical source manifest and an independent current-view check. A checksum detects an inconsistency; it does not establish authority.

The caller’s authorised scope is supplied separately to validation. A packet cannot widen that scope by declaring a larger allowance. Query interpretation and rewriting remain subordinate to the scope that authorised them. A retrieved instruction requesting another source supplies no additional permission.

Resolve evidence, not plausible identifiers

A selected row retains its source, document, version, representation, segmentation, passage identity and content digest, together with its exact span. The validator looks up that qualified identity in the canonical manifest. It checks the span and reconstructs the source text before admission.

Equal text from two authorities remains two evidence objects. Our constructed duplicate-source test admits both without merging their origins. A stale version is not silently replaced by a newer sentence, even if the newer sentence looks more useful. The application must make that changed-evidence decision explicitly.

Rank origins remain attached to the row. A BM25 score and a cosine score describe different scoring procedures; they do not become comparable confidence estimates because they occupy the same response. The contract preserves method, rank and score provenance without promising universal calibration.

Relationships carry a stronger obligation. A non-empty relationship path requires a trusted current-view validator. Without one, this application refuses the relationship rather than promoting a connected sequence of identifiers into supported evidence. The earlier graph implementation remains separately runnable; the downstream text experiment does not automatically integrate every earlier capability.

Recheck at admission

Provenance models help name dependencies but do not close the final evidence gap. The Open Provenance Model’s used edge records a declared dependency between a process and an artefact. It does not establish that a language model relied semantically or causally on a passage in its input. Likewise, our exact source spans identify origins, while execution records describe transformations and admission. Those records answer different questions, and this local contract claims no OPM conformance. Evidence use remains a separate assessment even when the provenance trace is complete.

The implemented sequence is:

contract = packet(execution, caller_sources)
admission = assemble(
    contract, canonical_units, collection,
    current_view, caller_sources, tokenizer,
    system_instruction, admission_budgets,
)

assemble first validates the contract and source references, then applies the existing projection against the supplied trusted view. In the ordinary run_request path, admission receives the request’s specified view and the same caller allowance used for retrieval. It does not independently obtain a later external authorisation state.

The revocation and version tests explicitly supply a changed trusted view to admission after selecting the original evidence. Given that updated state, the contract refuses newly ineligible or superseded references before prompt construction. The revoked-view demonstration exercises the same explicit configuration. These checks establish conditional refusal when updated state is supplied, not automatic detection of real-world permission changes. Live authorisation discovery is not implemented.

Only then does the assembler consider capacity. It constructs the actual chat input with the actual consumer tokenizer, measuring both the evidence JSON and the complete prompt. It admits whole units that fit and records capacity exclusions separately. It does not clip a source span while retaining the original reference as if nothing changed.

The two admission limits answer different questions. A selected-result allowance describes retrieval’s packet; a consumer-token allowance describes the input that can actually be sent. A selected unit may be perfectly valid and still fail admission. That is a capacity outcome, not a retrieval miss or proof that the source lacks the answer.

Return failures with useful meanings

The implementation has distinct internal error categories for invalid input, unavailable references or versions, unsupported relationships, exceeded capacity, unavailable consumers, consumer execution failure and unverified answers. An empty evidence packet has an explicit no-evidence admission status. It can still enter the deliberately declared closed-book condition; it is not relabelled as a successful grounded answer.

Contract area The promise it supports
Original request and query identity A search is accountable to the request that authorised it, even after an interpretation or expansion
Source, document, version, representation, segmentation, passage identity Evidence resolves to a named record; equal text from two authorities stays two evidence objects
Exact span and parent content digest The admitted text is the recorded text, not a similar-looking replacement
Schema version and unknown-field rejection Compatibility is conservative: a new capability needs an explicit validator, not silent acceptance
Internal error categories A failure keeps its meaning for diagnosis instead of collapsing into one status
Generic public errors Missing, restricted, stale and unresolved evidence are indistinguishable to a caller
Separate consumer status A model failure after a valid packet cannot inherit the packet’s success

Public errors reveal less than internal diagnostics. Missing, restricted, stale and unresolved evidence share a generic unavailable-evidence response. A public caller should not learn that a restricted intermediate exists from a refusal’s document name or count.

Consumer execution is another independent outcome. A valid retrieval packet can be followed by a model failure or malformed answer. The application retains the raw output and marks that later stage accordingly. It cannot erase the failure by returning the earlier retrieval-success status.

Test promises at their boundaries

The new constructed contract suite checks identity preservation, changed requests, unsupported schema versions, unknown fields, forged sources, wrong spans, modified bodies, duplicate references and non-finite scores. It also exercises revocation, stale versions, scope widening, whole-unit capacity, prompt limits, generation allowance and citations to material never admitted.

These checks establish behaviour on the declared fixtures. They do not establish production authentication, arbitrary-schema compatibility or safety against an attacker controlling the process and canonical manifest. The named views are trusted local inputs, as in the earlier chapters.

Compatibility is correspondingly conservative. Unknown top-level fields and unsupported schema versions are rejected. Adding a new capability requires an explicit validator and tests. A provenance or source-version change is not a harmless optional field when it changes the evidence a consumer would receive.

We now have testable promises about the evidence crossing the retrieval boundary. Defending the Retrieval Boundary asks which parts of those promises survive untrusted content and hostile requests, and which remain outside the experiment.

Capstone checkpoint: carry this contract into a tool

The capstone retains retrieval-contract-v1 as its evidence contract. A small retrieval-capstone-v1 transport wraps it with the named collection, snapshot, returned text and bounded execution-trace identity. The wrapper adds delivery fields; it does not invent another meaning for the source reference.

retrieval_open verifies a complete reference against current source text and eligibility before returning an exact span. Changed, missing, restricted and forged references share an unavailable response. Whole-parent expansion is explicit and fails when it cannot fit. Pi receiving that tool result establishes delivery at the tool interface, not actual model-token admission, semantic reliance or helpfulness. The final checkpoint in What We Can Now Prove executes this boundary. The earlier provenance questions and consumer-format failure remain separate from this integration.

References and further reading

Background and extensions for this chapter:

  • Define the Contract. Extends explicit fields and failure states into a model-program contract.