# Bounded Adaptive Retrieval: When to Stop
# Book: Retrieval From First Principles
# Chapter URL: https://programmer.ie/books/retrieval/21-chapter/
# Prose fingerprint (sha256, normalized): 9937494ffc2e7a518644b781b81ebebb6255336a36d0329c315d04b89a686d23
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

The question asks which programming language GIMP uses. The first lexical search retrieves entries about interpreted languages and two other programming languages. Those results are topically related, but none of the returned documents is judged relevant to this particular need. Finding the topic is not the same as finding the requested evidence.
The controller’s term-coverage check finds that the lexical results miss distinctive features of the request. With one retrieval call remaining, it runs the predetermined dense search using the original query. The GIMP version-history entry then reaches the selected evidence, recovering a known-positive document absent from the lexical prefix. The controller stops because its call allowance is exhausted—not because it has verified an answer. Retrieving one judged-positive document does not establish that the packet supports a correct response.
This is an executed case from the new external evaluation, not an invented illustration of what a controller might do. We need to account for three separate facts: the evidence available when the decision was made, the rule that permitted the extra call, and the judged outcome after it. A defensible stopping decision and a successful retrieval outcome are not the same claim.
Graph and Structured Retrieval ended with tools whose source-backed paths can be checked. We now place a small controller above the retrieval tools. Its job on the text route is narrower than finding an answer: after the first search, decide whether the available observations justify spending the one remaining permitted call.
The implemented text controller always begins with lexical retrieval. It inspects the returned evidence and either stops or makes one predetermined dense retrieval call with the unchanged original query. It does not generate follow-up queries, rewrite the request, choose freely among retrieval tools or run a language-model reasoning loop. In particular, it does not implement ReAct’s or IRCoT’s model-directed reasoning and action, or Adaptive-RAG’s learned pre-retrieval routing. Here, adaptive means that a bounded second action depends on the observed first result; it does not mean general autonomous retrieval.
Make the loop a contract A fixed two-stage workflow runs lexical search and then dense search regardless of what the first returns. This controller must make a decision after seeing the lexical evidence. If it always continues, it is merely a fixed workflow with an extra decision-shaped step. If it sometimes stops, we must examine both what those stops save and what relevant evidence they leave undiscovered.
The controller’s state retains the original query and its stable identity, the trusted eligibility view, interpreted features, available tools, actions already executed, returned references, assessed missing features, remaining budgets and an explicit stop reason. An observed passage, its exact reference and its query-term overlap are recorded separately from the controller’s assessment. The fact that a passage contains query terms can be checked; the conclusion that those terms make the evidence sufficient is an inference, not a property supplied by the retriever.
The implemented loop is deliberately small:
inspect request and remaining allowance -> choose a legal read-only action -> observe eligible results -> assess the evidence provisionally -> continue or stop -> select and recheck current evidence Related research moves the decision to different places. ReAct interleaves model-generated reasoning with actions. IRCoT derives later retrieval queries from intermediate reasoning. Adaptive-RAG uses a trained classifier to choose a retrieval strategy before searching. Our executed comparison uses none of these policies. A deterministic term-coverage proxy decides after lexical retrieval whether to stop or invoke one fixed dense search, so this chapter tests neither learned routing nor generated follow-up queries.
Start with an assessment we can inspect The controller first retrieves lexical candidates. It tokenises the original query with the existing tokeniser, removes a declared small stop list and, for each returned unit that could individually fit the content allowance, measures the fraction of retained query terms present. It also checks exact device-style identifiers rather than treating nearby-looking identifiers as equivalent. When the declared coverage threshold is met, the controller may stop provisionally; otherwise, if the budget permits, it runs dense retrieval with exactly the same query text.
This test is a sufficiency proxy, not a sufficiency verifier. It can expose a clear vocabulary mismatch, but a passage containing every query term might still concern the wrong entity, omit a condition, contradict another source or fail to answer the question. The controller records which terms matched and which it treated as missing so its judgement can be inspected. It cannot consult relevance judgements, target passages, answerability labels or a known answer at decision time.
The threshold was selected from two declared candidate settings on a separate policy-selection slice and fixed before the final comparison. This avoids selecting the reported setting from the final outcomes. It does not show that a coverage value of 0.8 means an 80% chance of sufficient evidence—or that the same threshold will transfer to unfamiliar queries.
The executable interface is in examples/retrieval_controller.py:
result = controller.run( request, policy="adaptive", threshold=0.8, ) The request supplies the original query, its stable identity and the trusted view. The interface rejects added judgement fields, preventing the ordinary call path from passing answer labels into the stop decision. This is a checked software contract for the example, not an operating-system sandbox or protection against arbitrary replacement of the implementation.
Give every continuation a finite cost The common allowance is at most two logical retrieval calls, six returned candidate rows, six distinct returned-unit reads and two controller iterations. The final selector can return at most four whole units containing 4,000 characters in total, separators included. The character cap was selected from development lengths before comparative outcomes were inspected. These limits make extra actions accountable; they do not impose an equal model-token or computation budget.
On the adaptive text route, lexical retrieval requests three rows. Only if the proxy continues does dense retrieval request another three; the controller then fuses the observed origin rankings using the existing RRF rule and packs whole units. A fixed lexical baseline may spend all six returned-row slots in a single search. Thus the policies share maximum allowances but not candidate depths, action counts or ranking procedures. Those differences must remain visible when interpreting the results.
A returned-unit read counts evidence that the controller materialises for inspection. It does not count every vector scored by exact dense search, every posting visited or the work of building an index. Logical calls, returned rows and materialised units are useful operational traces, but they are not measurements of end-to-end latency, CPU time or peak memory.
A structured contact request can take the existing source-backed path route when the required declared relationships are available. That route still checks exact bridge identities and current source evidence; it follows supplied relationships rather than extracting unknown ones. An ambiguous request can remain unresolved instead of forcing an entity choice. These specialised paths do not change the text controller’s fixed lexical-then-optional-dense decision.
Return a reason that means what happened Stop reasons describe why execution ended, not what has been proved about the answer. proxy_sufficient means that returned words passed a heuristic test. resource_budget_exhausted means that the remaining allowance could not support another action. no_new_evidence means a further attempt produced no new reference. None certifies exhaustive search, absence of evidence, factual correctness or a complete answer.
For the GIMP request, the first three lexical results contain no judged-positive document. Dense continuation contributes one, and the selected packet recovers one of the four known positives: recall is one quarter. The controller nevertheless stops for resource_budget_exhausted, not proxy_sufficient. Its nDCG@4 is approximately .172, compared with .479 for fixed dense and zero for fixed lexical. The extra search fixes one omission relative to the lexical prefix, but it does not establish sufficiency or outperform the fixed dense route.
The controller checks cancellation and its cooperative deadline before and after each bounded tool call. A synchronous search already running is not pre-empted by those checks. Tests cover cancellation, expired deadlines, single-call and single-iteration limits, repeated unsuccessful retrieval, and oversized tool responses. An unsuccessful attempt still consumes its applicable allowance; failure does not buy an unrecorded retry.
Before exposure, public projection rechecks the current view and distinguishes individually eligible units from a validated relationship path. If the path becomes invalid, the packet can state that the evidence is unresolved without revealing whether a protected source or restricted relation exists. Protected intermediate records and denial inventories belong to authorised offline diagnostics, not the public response. A generic public reason can preserve the boundary without fully diagnosing it for the caller.
Observe a useful decision and a premature stop The GIMP continuation is not representative of every decision. For a tango-composer request, the first lexical results contain the query vocabulary but no known-positive document. The overlap proxy therefore stops prematurely. Always continuing adds judged relevant evidence and reaches nDCG@4 of approximately .130; the adaptive packet scores zero. Fixed dense reaches .363 and fixed lexical also scores zero. The first search looked convincing to the proxy because of shared words, not because it met the judged need.
The comic-creator request illustrates the opposite risk. Forcing the additional search changes the final ordering and lowers nDCG@4 even though the selected number of judged-positive documents does not increase. Adaptive scores approximately .358, always continuing .329, fixed dense .219 and fixed lexical .452. An extra candidate source can improve discovery yet still disturb an otherwise useful ranking; more retrieval calls are not automatically better evidence selection.
Across 34 external final questions, the controller stops after its first action on 21. In retrospective comparison with always continuing, another call would improve nDCG on sixteen of those stopped requests, leave it unchanged on three and reduce it on two. Each early stop selects three units although nDCG@4 permits four; even when both policies are compared at three results, continuation produces twelve improvements, six ties and three losses. Eighteen stopped queries already contain at least one judged-relevant document. That is not evidence that the remaining relevant material, required relationships or answer conditions have been covered. The document judgements also do not specify which positives must appear together in a generated answer. These counts diagnose the stopping rule after the experiment; the live controller cannot consult them when deciding whether to spend a call.
The aggregate result is a quality loss, despite fewer calls than always continuing. Adaptive nDCG@4 is approximately .277; always continuing reaches .387. Fixed dense reaches .418 and fixed lexical .315. The fixed baselines each use 34 logical calls, adaptive uses 47, and always continuing uses 68. These action counts are not equal computation or elapsed-time measurements. The controller demonstrably responds to observations and honours its limits, but this term-overlap stopping rule discards too much judged ranking quality in the tested population. Chapter 24 compares this outcome with the other baselines; the successful GIMP continuation cannot overturn the aggregate finding.
The recorded comparison. Fewer calls are not a quality gain: adaptive stopping returns the lowest judged ranking quality of the four policies. The second panel shows why the verdict on those stops depends on the cutoff, because each stop selects three units while nDCG@4 permits four. Missing evidence is a different obligation The fictional validation inventory contains no battery-replacement procedure for the requested need. The controller searches twice, returns related material and stops when its allowance is used. That is a bounded search outcome, not proof that no procedure could exist elsewhere. Another constructed request has the reset instruction but lacks a second required procedure. Recovering the available instruction does not satisfy the complete multi-part need.
These are genuine absence cases within a fully specified fictional inventory, judged by the same author who designed them. They expose how the controller behaves with missing requirements; they do not establish reliable abstention on unfamiliar collections. A negative note may be useful evidence for a different question, and returning it does not by itself constitute a false answer. No answer was generated or evaluated in this experiment.
We can now inspect a bounded decision, the evidence it saw, the cost it incurred and the cases in which its stop was premature. Judging the quality of those decisions required relevance labels that the controller itself was not allowed to see. Relevance Judgments and Test Collections examines the next layer of the evidence chain: who supplied those labels, what exactly they certify and which conclusions the evaluation cannot support.
References and further reading Background and extensions for this chapter:
Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR; camera-ready author manuscript. Interleaves model-generated reasoning and actions, unlike this chapter’s deterministic stop-or-continue policy. Trivedi et al. (2023). Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. ACL. Forms later retrieval queries from intermediate reasoning, whereas this chapter’s dense continuation reuses the original query. Jeong et al. (2024). Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. NAACL-HLT. Trains a classifier to route each question to no, single-step or multi-step retrieval before any evidence is seen; a different decision from this chapter’s post-retrieval stop. Related chapters Runtime State, Progress, and Termination. Extends stopping rules into explicit progress, state and termination contracts.