Part III — Retrieval Is an Experiment

“Just retrieve the relevant documents”

There is no such operation. What actually runs is a pipeline, and every stage has parameters that change the output:

    flowchart TD
    Q[query] --> QT["query transform — raw / rewrite / expand / HyDE / decompose"]
    QT --> RE["representation — which model, normalization, chunking"]
    RE --> CG["candidate generation — ANN index, nprobe / ef, pre-filters"]
    CG --> SI["similarity — metric, hybrid dense+sparse weighting"]
    SI --> TH["threshold — minimum score to survive"]
    TH --> RK["ranking — rerank top-N with a cross-encoder?"]
    RK --> DV["diversity / dedup — MMR, cluster collapse"]
    DV --> TK["top-k — how many survive"]
    TK --> CA["context assembly — order, dedup, budget, citations"]
    CA --> M["what the model actually sees = the system's memory"]
  

Retrieval does not return “the relevant documents.” It returns whatever this specific chain of decisions produces — and that is the system’s memory.

Each link is a lever with a failure mode

• Chunking. Too large: one chunk dilutes the query match and wastes budget. Too small: the answer spans two chunks and neither retrieves. Chunk boundaries are a retrieval parameter, not a preprocessing detail.
• Query transform. Expansion raises recall and lowers precision. Decomposition helps multi-hop questions and hurts simple ones. HyDE helps when queries are short and unlike the corpus.
• Hybrid weighting. Dense catches paraphrase; sparse (BM25) catches exact terms, names, codes, rare tokens. The blend weight is a tuned parameter; pure dense loses on entity-heavy queries.
• Threshold. The difference between “return the best of a bad lot” and “return nothing, correctly.” Set from score distributions (Chapter 14), not by eyeballing.
• Reranking. A cross-encoder over the top 50–200 fixes some of Chapter 10’s failure types. Costs latency. Worth it exactly where the distractor probe said rerank.
• Diversity. Without it, k slots fill with near-duplicates. With too much, you drop the second-best true answer.
• Context budget. More retrieved passages is not more knowledge; it is more tokens, more distractors, and the lost-in-the-middle effect. This is the Context book’s territory, entered here.

The policy is the artifact

A retrieval system’s behavior is fully described by its policy — the chain above with every parameter bound — plus the space records of the models it uses. Two teams “using the same embedding model” can have entirely different retrieval behavior because their policies differ. The policy, not the model, is what you version, test, and roll back.

Demonstration: same corpus, three policies

MEASURED on RELATE v0.1, Wave 1 row 1.13 — artifact experiments/embeddings-from-first-principles/wave1/artifacts/policy-ablation.json. Model all-mpnet-base-v2 held fixed.

RELATE queries, one embedding model held fixed, three policies:

policy                                                     nDCG@10   hard-neg nDCG@10
A: dense only                                                0.952         0.958
B: hybrid (0.6 dense / 0.4 BM25)                             0.926         0.929
C: B + NLI cross-encoder rerank of the top 50                0.856         0.929

MEASURED — on RELATE v0.1, policy layering did not help; it hurt. Adding BM25 injected lexical noise into an already-solved ranking, and a generic NLI cross-encoder damaged it further (the same mismatch seen in Chapters 10 and 15). This is the ceiling effect again: RELATE’s queries are near-restatements of their answers, so plain dense retrieval already saturates and there is no hard-case quality left for the policy layer to recover. The chapter’s claim — that the policy, not the model, carries most of the achievable quality and latency on hard cases — is well supported by the retrieval literature (BEIR hybrid results; cross-encoder reranking gains on MS MARCO), but demonstrating it needs queries with a real gap to the answer (logged for RELATE v0.2). The structural point stands: “we use model X” describes one stage of eight.

What this chapter establishes and what it does not

Establishes: retrieval is a multi-stage policy; each stage has parameters with distinct failure modes; the policy (not the model) is the versioned, tested artifact; large quality and latency differences come from the policy layer.

Does not establish: an optimal policy (workload-specific), or that the embedding model does not matter (it sets the ceiling). It establishes that “we use model X for retrieval” is not a description of a retrieval system.

Lab 12: policy ablation

PROPOSED, not executed.

Setup. Fixed corpus, fixed embedding model, 200+ labeled queries including a hard-negative subset.

Task.

1. Baseline policy: dense, k=10, no threshold, no rerank. Score.
2. Add one stage at a time: hybrid → threshold → rerank → diversity. Score after each. Record latency.
3. For chunking: re-index at 2 chunk sizes; score.
4. Identify the single stage with the largest hard-negative gain per millisecond.

Success criterion. A ranked list of your policy stages by hard-negative quality gain, and a stated latency budget with the policy that fits it.

Companion component: the policy object

retrieval_policy:
  id / version:      <string>
  spaces:            [space_record, ...]      # embedding + reranker
  chunking:          {size, overlap, boundary_rule}
  query_transform:   <...>
  candidates:        {index, ef/nprobe, prefilter}
  similarity:        {metric, hybrid_weights}
  threshold:         {value, source: "Ch14 calibration on set S"}
  rerank:            {model, top_n} | none
  diversity:         {method, lambda} | none
  top_k:             int
  context_assembly:  {order, dedup, token_budget, citations}
  eval:              {nDCG@10, hard_neg_nDCG@10, p95_latency, eval_set}

The Observatory treats this object as the unit of change: every deployment is a policy version with attached eval numbers, and rollback restores a whole policy.

Failure modes

• “We use model X.” Names one stage of eight. Not a system description.
• Tuning the embedding model while the policy is untuned. The larger, cheaper wins are usually in reranking and hybrid search.
• Chunking as preprocessing. It is a retrieval parameter; ablate it.
• Threshold by intuition. Use the score distributions (Chapter 14).
• Maximising k. More passages = more distractors and more tokens, not more knowledge.

What this chapter established

• Retrieval is an eight-stage policy from query transform to context assembly; each stage has parameters and a failure mode.
• The policy, not the embedding model, is the artifact you version, test, and roll back.
• On RELATE v0.1 (row 1.13) policy layering did not help — its near-restatement queries leave no hard-case headroom; the literature shows hybrid + reranking gains on harder retrieval sets, and the structural point (the policy is eight stages, the model is one) holds regardless.
• The policy object: every parameter bound, every stage’s eval attached, rollback restores the whole policy.

Next

Part III measured retrieval outcomes. Part IV steps back to the representation itself and asks how to evaluate it — separating representation quality from application quality, and confronting how little a generic similarity benchmark tells you about your problem.