← Retrieval From First Principles

Learned Sparse and Multi-Vector Retrieval

Inspect learned vocabulary weighting and late interaction without mistaking hand calculations for trained retrieval results.

The user asks how to purge a cache. BM25 can miss the command because the query and instruction use different words; dense retrieval can find the topic yet rank a general overview above the executable step. Hybrid Search and Reciprocal Rank Fusion combines the resulting lists, but combining ranks cannot restore distinctions that neither representation made useful to its scorer.

The next question is what a representation preserves for matching. A single weighted word list and a single sentence vector discard different details. Learned sparse retrieval keeps vocabulary dimensions but learns their weights, including possible expansion terms. Multi-vector retrieval keeps several contextual vectors for a unit, allowing comparisons at a finer granularity than one whole-unit score. Both change what the retrieval stage can notice; neither changes the underlying source evidence.

Three kinds of evidence must remain separate in this chapter: published descriptions of trained systems, hand-written examples whose scoring functions we executed, and results measured on this book’s retrieval collections. Only the hand calculations were executed for learned sparse and multi-vector matching. Their actual trained models were not measured: no suitable cached model was available, and downloading a large model was not made a condition of completing the batch.

Evidence kind In this chapter
Published description The SPLADE and ColBERT papers, with the authors’ repositories, for how the mechanisms are defined and implemented
Executed by us Sparse-dot arithmetic and MaxSim on hand-written vectors; two frozen controls, no trained model run
Measured on this book’s collections Nothing for learned sparse or multi-vector retrieval: accuracy, index size, latency and memory remain not measured

A sentence may quote the first row, the second or the third, but never two at once. The rest of the chapter labels each claim the same way.

Keep vocabulary dimensions, change their weights

BM25 computes a term contribution from occurrences, collection rarity and passage length. Learned sparse retrieval keeps a vocabulary-addressed representation, but a model decides which dimensions receive weights. It may weight words already present or activate expansion dimensions that were not literal words in the unit. The index can still exploit sparsity for term-oriented lookup, but those learned weights must not be read as BM25 frequencies or IDF values.

In the original SPLADE paper, a masked-language-model head generates vocabulary scores from contextual token representations. Positive activations are log-saturated and combined across positions, with sparsity regularisation limiting how many dimensions become active. The original paper uses a sum across positions; later variants can use other aggregation rules. Thus a label such as “SPLADE” does not by itself specify the encoder or the resulting scores. The authors’ implementation provides a primary record of implementation choices.

We can illustrate how those weighted dimensions are compared without claiming to run SPLADE. The following values are written by hand, not emitted by a trained encoder:

query = {"cache": 1.0, "purge": 0.5}
document = {"cache": 0.7, "purge": 0.2}
score = sparse_dot(query, document)
# 1.0 * 0.7 + 0.5 * 0.2 = 0.8, within floating-point precision

sparse_dot multiplies weights only for vocabulary keys shared by the query and document, then adds those products. Missing dimensions contribute zero. If a trained encoder activated purge for a document whose wording used another expression, it could make a previously impossible lexical match possible. Here we supplied that dimension ourselves. The numerical result tests sparse-dot arithmetic, not whether learned vocabulary expansion works on this query.

A sparse dimension is not necessarily a quotation from the document. In a learned representation, purge may be weighted even when the source never says “purge.” That feature can help find the document, but it is not source evidence to quote. The result must still carry its original text and exact reference; the encoder’s vocabulary weights must never be substituted for a cited passage.

To reproduce such a result, the origin record would need more than a model name: exact vocabulary and tokeniser, model revision, positional aggregation, any pruning rule, scoring convention and the complete retrieval-unit reference. Pruning can remove a dimension that would have admitted a candidate, changing the pool before fusion begins. That would require another controlled comparison, not merely a renamed generator in the trace.

Delay interaction until token representations meet

A single-vector retriever reduces each query–unit pair to one vector comparison. A multi-vector retriever retains several contextual representations of a unit and defers some matching decisions until the query and document vectors are compared. The additional structure offers more possible matches, at the cost of more representation and search work.

ColBERT’s original late-interaction mechanism separately encodes query tokens and document tokens. For each query vector, MaxSim finds the highest similarity to any document vector; the final score adds those per-query maxima. The same document vector may be chosen for several query vectors, so MaxSim is neither one-to-one alignment nor a demonstration that distinct evidence spans satisfy the question. Tokenisation, projection and other details depend on the model and implementation; the authors’ repository documents the executable system.

A two-dimensional hand example shows what this scoring rule can distinguish. The query looks in two perpendicular directions. One document keeps those directions separate; the other offers a single compromise vector:

query = [[1, 0], [0, 1]]
separate = [[1, 0], [0, 1]]
collapsed = [[1, 1]]

maxsim(query, separate)   # 1 + 1 = 2
maxsim(query, collapsed)  # 1/sqrt(2) + 1/sqrt(2) = sqrt(2)

Now mean-pool each list and normalise it to a single vector. In this constructed geometry, both documents point in the same direction as the pooled query, giving cosine similarity one for each. MaxSim instead rewards the document that preserves two separately matching directions: the frozen control scores it at 2, against approximately 1.4142 for the compromise vector. The difference is what the scoring representation can express, not a measured difference in answer quality.

Neither document vector in this example comes from language-model training. The calculation proves only that pooling can erase a distinction retained by MaxSim. It does not show that ColBERT retrieves better evidence on our queries. Even a large MaxSim score may aggregate attractive token matches that do not form the relationship, instruction or explanation the user needs.

More retained structure creates new costs

A learned sparse index needs weighted vocabulary entries and postings; a multi-vector index may retain many vectors for each retrieval unit. More retained structure changes construction, pruning, candidate search and storage requirements, and often shifts work between indexing and query time. The size of that trade-off depends on the trained model, unit lengths and implementation. We have not measured learned-sparse or multi-vector retrieval accuracy, index size, latency or memory in this chapter.

Three columns comparing representations of one unit. Learned sparse keeps weighted vocabulary entries such as cache 0.7 and purge 0.2. Multi-vector keeps one contextual vector per token and scores with MaxSim. Single vector pools the unit into one 384-dimension vector, averaging directions before scoring. A footer records that 3,681 of 5,183 SciFact inputs exceed the pinned encoder's 256-wordpiece limit.
Three amounts of retained structure for the same unit. The vocabulary weights and token vectors shown here are hand-written teaching values, labelled inside the figure; the input-limit audit in the footer is a recorded count from the pinned Part III run.

Three lengths need separate records: the text stored as the retrieval unit, the portion actually consumed by the encoder, and the features or token vectors retained for scoring. A final evidence packet can expose the complete source slice even though its score was calculated from truncated or pruned information. Preserving the full packet does not undo that representation loss; the trace needs enough metadata to reveal it.

The single-vector MiniLM encoder actually executed earlier truncates input at 256 wordpieces. In the external collection, 3,681 of 5,183 document inputs exceed that limit. Those counts establish potential input loss for that pinned run, not which individual ranking errors truncation caused or whether a richer representation would repair them. Learned sparse and multi-vector models would require their own input-limit, retained-feature and cost audits.

Chapter 11’s complete-reference union could accept another generator operating over the same declared retrieval units, recording its own model and scoring provenance. If a new method creates different units, matching text or sharing a parent document is not enough to merge references: unit mapping and judgement semantics must be defined first. Both paths remain subject to eligibility and current-view projection. More sophisticated matching features do not grant permission.

Choose the next experiment by the missing information

The next experiment should target the information that was actually lost. When mismatched vocabulary prevents a useful candidate entering the pool, learned expansion is one hypothesis. When a whole-unit vector blurs distinct aspects, late interaction is another. We have illustrated how both mechanisms score representations, but have no frozen trained-model comparison showing that either solves those failures.

The existing results give those hypotheses different targets. SciFact claim 1368 has no judged-positive document even in the largest candidate pool, so another candidate generator might be worth evaluating. Claim 873 already has judged-positive candidates in its pool but loses them before the final cutoff, making improved ordering a different possible intervention. Neither case verifies that a learned sparse model, multi-vector model or reranker would repair it.

The second intervention can be tested with models and candidates already available locally. Rerankers and Cross-Encoders compares a query jointly with each existing candidate, while holding the candidate pool fixed. That experiment can test ordering; it cannot establish what a new sparse or multi-vector generator would have retrieved.

References and further reading

Background and extensions for this chapter: