Retrieval From First Principles on Programmer.ie: Modern AI programming

The Information Selection Problem

Fri, 16 Oct 2026 00:00:00 +0100

An engineer asks how to restore a device to its factory settings. The fictional handbook contains the instruction, “R17 reset: hold the blue button for ten seconds to erase all setup.” Yet a search for “restore factory settings” returns nothing. The answer-bearing passage is present; the search rule cannot connect the engineer’s words to the handbook’s.

The engineer tries “R17 reset” instead. This time the search finds both the device handbook and a network handbook. Their scores tie, so an identity-based tie-break puts the device instruction first. The order is reproducible, but the query has not established which operation the engineer intended. A deterministic result can still be the wrong result.

Documents, Identity, and Provenance

Fri, 16 Oct 2026 00:00:00 +0100

The sentence is correct. The attribution is wrong.

A fictional specification says, “R17 reset: hold the blue button for ten seconds.” An old blog repeats those words exactly. Suppose we key our document records only by a digest of their text and keep one metadata record per digest. The blog is inserted second, so its source metadata replaces the specification’s. The quotation survives unchanged; the distinction between its two origins disappears.

Building an Inverted Index

Fri, 16 Oct 2026 00:00:00 +0100

We search for cache reset, then repeat the search. Our first retriever reads every eligible passage and tokenises it each time. The documents have not changed, but we do the same preparatory work again.

The scan made matching easy to inspect, but repeats work when queries change more often than documents. An inverted index prepares the term-to-passage relationships once, so each query can consult the relevant lists instead of rereading every passage. The improvement must preserve the evidence contract from Documents, Identity, and Provenance: faster discovery is no use if a result loses its source, document or version identity.

BM25: Ranking with Terms

Fri, 16 Oct 2026 00:00:00 +0100

Two passages match cache reset. One is a long maintenance note that mentions the topic repeatedly. The other gives the command the user needs:

To reset the cache, run cache-clear.

Both passages contain the two distinct query terms, so our overlap scorer ties them. The identity-based tie-break puts the verbose note first. That order is reproducible, but nothing about the tie-break establishes that the note better satisfies the request.

Building an Inverted Index gave us candidate passages, term frequencies and lengths without repeatedly scanning their text. Now we need a ranking rule: when passages share query terms, what should make one outrank another? BM25 uses those retained statistics to answer that limited question. It does not determine whether a passage is true, permitted or sufficient for the task.

The First Search Baseline

Fri, 16 Oct 2026 00:00:00 +0100

The command passage now ranks first for cache reset. That is the result we designed the Chapter 04 example to illustrate. It shows how BM25 can change an ordering; it does not yet show that BM25 is generally the better ranking method.

To evaluate that claim, we need information needs and relevance judgements fixed before inspecting the comparison. We also need to hold the searchable collection and candidate policy steady while changing the ranking mechanism. Otherwise, we can keep choosing examples that favour whichever method we have just built.

Parsing, Chunking, and Retrieval Units