# Latency, Cost, and Operational Reliability
# Book: Retrieval From First Principles
# Chapter URL: https://programmer.ie/books/retrieval/25-chapter/
# Prose fingerprint (sha256, normalized): 482f13af0f4641ffca3cb00be0ea0f0d055b28cdd7773b66660e1be7ce2d96db
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

For a request on which the adaptive controller stops after its first lexical search, it and the fixed dense baseline each record one logical retrieval call. But one call retrieves term matches and the other scores vector candidates. On the same desktop, those operations have very different measured latencies in our reference implementation. The dense scan uses Python loops and copied unit records, so its measured cost reflects this implementation as well as the retrieval method; it is not an intrinsic latency of dense retrieval.
A call count tells us whether a procedure stayed within its declared allowance. It does not tell us how much computation, preparation, memory traffic or elapsed time that allowance consumed.
Controlled Experiments and Regression Gates showed that adaptive stopping makes fewer retrieval calls but achieves lower average judged quality. A cost argument must now answer three separate questions: which work did those calls represent, which work was done before timing began, and is the measured trade-off acceptable for a specified workload?
Draw the timing boundary first Retrieval work crosses several timing boundaries: source preparation, index construction, model construction, document encoding, query encoding, candidate search, controller assessment, reranking, selection and current-view projection. A number labelled simply “latency” conceals whether it includes those stages. Before comparing policies, we need to say exactly when the clock starts and stops.
Our repeated query measurement surrounds the actual controller call:
start = time.perf_counter() result = controller.run(request, policy, threshold) elapsed = time.perf_counter() - start This interval includes the controller’s decisions, executed ranking calls, fusion, whole-unit selection and the final current-view check. It excludes index construction, model construction, and query and document encoding; the query vectors have already been prepared. The result is warm-index, cached-vector query latency. It is not the elapsed time a new user would experience if their query still needed encoding or the system first needed to load its models.
Python’s performance-counter contract provides a clock for elapsed-time differences. The programmer still determines what work falls between the two readings. The experiment therefore records the machine, interpreter, packages, thread settings and timer properties as well as the durations shown to the reader.
Repeat a declared workload We selected four timing questions in final-query order before inspecting the final outcomes. For each policy, one warm-up precedes seven timed trials, with policy order rotated across trials. The indices and vectors are shared throughout, but ranked results are not cached. We also leave the operating-system file cache warm rather than flushing it. This is a repeatable local workload, not a cold-start or production-load experiment.
For each of the four questions, we take the median of that policy’s seven timed trials. The table gives the minimum and maximum of those four per-query medians. It does not show the fastest and slowest individual trial, a confidence interval or a production tail-latency percentile:
Policy Range of query medians Lexical 18.3–133.6 ms Dense 190.3–228.8 ms Hybrid 206.5–342.7 ms Adaptive 16.6–130.1 ms Always continue 212.8–316.5 ms DenseIndex.rank in examples/semantic_search.py iterates over candidate records, deep-copies each permitted record, computes dot products with Python scalar operations and sorts the results. A vectorised matrix–vector scan could have substantially different costs, but we have not measured one. The table compares these executed Python paths; it cannot establish the relative latency of dense and lexical retrieval in optimised services.
All four timed questions take the adaptive controller’s first-action stop branch. The timing panel therefore measures early stopping and the additional cost of forcing continuation on those same questions. It contains no timed example of the adaptive controller taking its second action. Its ranges must not be read as the latency distribution of all adaptive requests or as an average over the 34-question final evaluation.
On one album-related request, adaptive median latency is about 17.6 ms and hybrid median latency about 206.5 ms. The additional search also improves judged quality on that request, so the saved time comes with a measurable evidence trade-off. On the comic-creator request, continuing instead lowers nDCG; on the racing-driver request, both policies have zero nDCG. A decision to continue must use evidence available at decision time, not the relevance labels used retrospectively to explain these differences.
On this timing slice, the router, hybrid and always-continue policies select identical evidence, yet their measured times differ. Those differences do not show that one policy label causes faster execution when the underlying work and outputs are equivalent. Seven trials reveal local variability, not a stable deployment ranking, concurrent throughput or a 99th-percentile service guarantee.
Account for preparation honestly The cost capture records about 1.00 seconds to construct the lexical index and .415 seconds to construct the dense-index object from vectors already retained. Both measurements start after earlier source loading and collection preparation. The dense-index figure in particular does not include encoding the documents. Neither number is a complete ingestion or first-query startup cost.
The pinned local encoder takes about 117.98 seconds to encode the new batch, excluding model construction and token-length inspection. The batch includes 6,088 evidence-reference inputs and 64 query strings, deduplicated to 6,135 distinct strings. That is one combined workload, not a measured document-only cost. Nine retained unit references exceed the encoder input limit; this identifies a representation boundary, not nine demonstrated retrieval failures or a tested repair.
A separate probe encodes the eight development queries together as one batch. After warm-up, its median across seven trials is about 39.6 ms. The batch timing cannot be divided by eight and presented as single-query interactive latency: per-batch overhead and execution behaviour make that a different operation.
Model construction is recorded separately using cached local weights. The constructor in the completed encoding run takes about .174 seconds; a later cost process has its own recorded constructor duration. Neither observation includes imports or a genuinely cold filesystem. The cost probe also does not perform a second full corpus encoding, so it cannot serve as a second independent measurement of that batch.
When a corpus and model remain stable, their preparation cost can be spread over requests that genuinely reuse them. The accounting below separates work performed for each request from setup shared across requests. It is a model for thinking about cost, not a measured per-request total:
per-request work = query encoding + searches + control + projection amortised setup = relevant preparation cost / requests sharing that preparation The following table keeps each measurement beside the work excluded from its clock. The figures should not be added together indiscriminately: some cover combined batches, others isolate object construction or a warm query, and their reuse assumptions differ. None is a measured complete ingestion cost or end-to-end request latency.
Recorded measurement Value What the boundary excludes Policy latency, seven trials after warm-up lexical 18.3–133.6 ms; dense 190.3–228.8 ms; hybrid 206.5–342.7 ms; adaptive 16.6–130.1 ms; always continue 212.8–316.5 ms Index build, model construction, query and document encoding Lexical index construction about 1.00 s Earlier source loading and collection preparation Dense index object over retained vectors about .415 s Earlier source loading; it reuses retained vectors Pinned local encoding of new inputs about 117.98 s for 6,088 evidence inputs plus 64 query strings, deduplicated to 6,135 strings Model construction and token-length inspection Query-encoding probe on one batch of eight median about 39.6 ms over seven trials after warm-up Anything interactive: batching changes the operation Cross-encoder six-pair batch median about .170–.284 s across the four timing questions Model construction, token-length inspection and the warm-up Model construction from cached weights about .174 s in the completed encoding run Imports and genuinely cold filesystem behaviour Any amortised estimate must identify both the preparation being shared and the number of requests that can actually reuse it. Dividing the combined encoding batch by an assumed traffic volume would create a hypothetical allocation, not a measured per-request cost. Document-only preparation was not isolated, and a changed corpus version or model fingerprint may invalidate some of the supposed reuse.
Measure another stage without claiming a quality gain The isolated reranking probe runs the pinned cross-encoder on the first six lexical candidates for each of those same four timing questions. It records the actual query–document pairs, input digests, lengths, truncation flags and raw scores. The main retrieval policies never consume these new scores. This isolates the cost of an additional scoring stage without turning it into an untested quality intervention.
Across the four questions, median prediction time for a six-pair batch ranges from about .170 to .284 seconds. Model construction, token-length inspection and warm-up fall outside this interval. The measurement tells us what scoring those six supplied pairs costs in this local setup. It cannot show that reranking would improve selection, and a reranker still cannot score a useful document that never entered its candidate pool.
No paid model calls or API-token charges were incurred, and we do not assign fictional commercial prices. The absence of a bill does not mean local inference is free: it consumes time, hardware capacity and energy. Only the operations actually recorded here are quantified.
Keep size separate from memory The float32 vector payload occupies 9,449,472 bytes. The diagnostic lexical-index JSON representation occupies 19,919,337 bytes and includes readable field names, copied source text and postings. These describe two different stored representations, not comparable measurements of the complete running systems.
Neither byte count measures resident process memory. The running system also holds Python objects, copied records, allocator overhead, model weights and temporary tensors. Peak and resident memory are not measured. We therefore cannot infer a safe memory limit from either stored payload size.
The same distinction applies to evidence consumption. Adaptive selects fewer body characters on average than hybrid but also recovers less judged evidence. The character cap excludes citation wrappers and other prompt material, so it is not a measured language-model token budget. Candidate rows, selected source characters, encoder inputs and final consumer context must be accounted for separately.
Preserve reliability at the cache boundary A reusable vector needs an exact binding to the text and model that produced it. A reusable index needs an identity for the collection view it indexed. Otherwise an old source body, changed version or wrong query envelope may appear compatible while producing stale or ineligible evidence. The tests exercise such provenance mismatches and current-view revocation rather than relying on document names alone.
The named-view interface assumes that a trusted caller supplies the applicable policy view. It neither discovers live permission changes nor authenticates production users. The final bounded projection rechecks selected references under that supplied view, but it does not make every upstream cache safe for arbitrary shared use. Cached search results and current disclosure permission remain separate decisions.
The controller checks a cooperative deadline before and after synchronous tools and records the applicable stop reason. It cannot forcibly interrupt a tool already executing. Since this implementation has no external search service or pre-emptive cancellation mechanism, it establishes no guarantee about aborting a stalled remote call. Retries, concurrent load, production cache invalidation and operational service levels remain unmeasured.
Choose a trade-off rather than a slogan On the full judged-quality slice, adaptive makes 47 calls against 68 for the two-stage workflow, but its mean recall and nDCG are lower. Fixed dense achieves higher reported mean quality than adaptive with fewer logical calls. In the four-query timing panel, however, an early lexical stop is faster than this particular Python dense-search path. Those statements concern different measurement scopes: full-slice quality and call counts versus local timing of one controller branch. Combining them into a universal speed–quality ranking would overstate the experiment.
A Pareto claim requires a named workload and dimensions: one system must be no worse on the specified quality measures and no more costly on the specified measured resources, with at least one strict advantage. The local early-stop times do not establish latency dominance across all queries or against vectorised dense search. Likewise, higher mean nDCG alone does not settle an application’s latency allowance. We keep quality, latency, selected text, preparation and reliability visible as separate quantities rather than inventing one overall winner.
Together, these five chapters give us an auditable retrieval decision process and an equally important account of its limits. We can observe when it searches, why it stops, what evidence survives selection and which costs were actually timed. But retrieval quality remains one stage short of a task outcome. Does Better Retrieval Improve the Task? crosses that boundary with a real consumer, asking whether selected evidence reaches its prompt and whether the resulting output meets the task contract. Its consumer measurements must not be confused with the retrieval and operational measurements reported here.
Capstone checkpoint: inspect a decision and its work python -B -m examples.capstone.milestones 5 --summary This executes a small, prospectively declared teaching task set using lexical, hybrid and the existing .8 stop-or-continue proxy. The summary reports actual calls, candidate rows, stop reasons and defined document-level recall for the three policies; it also shows selected counts for tasks with no positive labels. Full per-query identities, metrics and exact-reference checks are stored in .artifacts/capstone/runtime/milestones/05.json; omitting --summary preserves the original JSON status. The labels are same-author teaching judgements, unjudged evidence remains unknown, and queries with empty positive sets have undefined recall. This milestone does not prove general superiority or answer quality. The batch shares one loaded encoder, while a one-shot Pi tool call incurs different startup work. Source loading, scorer reconstruction, model construction and cache writes still matter even when a report counts only logical calls.
References and further reading Background and extensions for this chapter:
Dean and Barroso (2013). The Tail at Scale. Communications of the ACM. Extends operational measurement from average latency to variable components and service tails. Johnson, Douze and Jégou (2017). Billion-scale similarity search with GPUs. arXiv preprint, v1. Examines the interaction between vector-search algorithms, hardware and memory access. Broder et al. (2003). Efficient query evaluation using a two-level retrieval process. CIKM. Provides an explicit candidate-screening mechanism for reducing full scoring work. Related chapters Reuse Before Recompute. Extends reuse and caching into context cost and recomputation decisions.