Hybrid Search and Reciprocal Rank Fusion

The evidence is already in the candidate pool, yet the final ranking makes it harder to find. For SciFact claim 1337, BM25 ranks the annotated document first. With ten candidates per generator, Reciprocal Rank Fusion (RRF) moves that document to second place; the later cross-encoder also leaves it second, with reciprocal rank one half. Nothing was lost from the pool. The loss occurred when the candidates were ordered.

Candidate Generation as a Pipeline showed that combining lists can enlarge the pool of potentially useful evidence. But a larger pool is only an opportunity: its ranking can still hide the evidence we wanted to preserve. This chapter asks how to combine unlike rankings with an inspectable rule—and whether the resulting order actually improves judged retrieval.

Start with incompatible measurements

BM25 scores depend on term matches, collection frequencies and passage lengths. Dense cosine scores describe a different quantity: the alignment of normalised vectors. A BM25 score of seven and a cosine score of .7 do not mean that one result is ten times as relevant as the other. Adding them directly would make their incompatible scales, rather than an explicit decision about relevance, determine the combined order.

Instead of combining score magnitudes, we can combine ranks. A document ranked first by BM25 and a document ranked first by dense retrieval then receive the same first-place contribution, regardless of their raw scores. This removes the problem of incompatible scales, but also discards score gaps: a decisive lead and a near tie are treated alike. Rank fusion changes what information controls the result; it does not calibrate relevance.

Cormack, Clarke and Büttcher’s RRF paper combines contributions that diminish with each candidate’s position in a ranking. To reproduce the experiment, we must specify more than the formula: which lists are supplied, how far they extend, how missing items contribute and how ties are resolved. Our rule is:

RRF(d) = sum over supplied lists containing d of 1 / (60 + rank(d))

Each supplied list uses one-based ranks. A candidate absent from a list contributes zero; a candidate appearing in both lists receives two contributions. We fuse BM25 and exact dense rankings with equal weight after truncating each to its declared depth. The denominator’s constant, sixty, was fixed before final evaluation; we did not choose another value after inspecting the results.

The list protocol is part of the scoring definition. Cormack and colleagues formulate RRF over complete ranking permutations. Bruch, Gai and Ingber (§5.1) describe approximating unknown ranks by ranking within the candidate union. Our implementation instead preserves each supplied list’s original positions and assigns zero contribution where a candidate is absent. Two experiments can use the same reciprocal-rank expression and still produce different rankings because they treat unseen positions differently.

Work a small example before running a benchmark

Consider two short rankings: lexical search returns A then B, while dense search returns B then C. A, B and C stand for complete evidence references, each tied to a particular source and version—not merely interchangeable text strings. The contributions are:

B receives credit from both lists and therefore outranks A and C, which appear only once. The executable control retains each contribution, and focused tests check the arithmetic, missing-list cases and deterministic ties. That agreement says nothing yet about whether B is relevant. Nor may equal wording collapse A and B if they refer to different sources: fusion combines rankings of references, not unidentified strings.

The implementation first constructs the union of complete references, sums contributions from their recorded generator ranks, and sorts by descending fusion score. Equal scores are broken by the complete reference tuple, not by set iteration, dictionary insertion or incidental model output order. A reference cannot appear twice in one generator list and gain a second contribution: such duplicates are rejected.

The reference implementation retains one comparison rule with the constant set to zero. Under that rule, rank one contributes 1 and rank ten contributes .1. With the constant set to sixty, those contributions are much closer, so agreement across lists can weigh more heavily relative to the gap between positions. The zero-constant control is still a sum of reciprocal ranks, not a sum of BM25 and cosine scores. It was declared as a control, not selected as a replacement after inspecting the final sample.

The original RRF paper considered zero in a parameter-sensitivity analysis and reports choosing sixty in a pilot. Rank fusion avoids direct score normalisation, but it is not parameter-free. The constant, generator depths, handling of missing candidates and tie convention all form part of the retrieval policy.

Truncated lists define the experiment

A depth-ten generator list excludes the candidate at rank eleven. Our RRF calculation therefore gives it no contribution from that list, even if a deeper search would have found it. Increasing generator depth can introduce new candidates and change the order of candidates already present. Zero contribution means not supplied by this list, not judged irrelevant.

The external comparison keeps the eligible document collection, title-and-abstract representation and sampled queries fixed across methods. Each generator supplies up to ten candidates; the reported metrics evaluate only the first three ranked documents. The frozen results are:

The ranking averages cover the twelve sampled SciFact claims with annotated positive documents. The other eight claims have no judged positives, so their relevance metrics are undefined and their returned candidates are reported separately. As required by these conventional metrics, documents outside the annotated positive set receive zero gain in the calculation; this does not establish that every such document has been independently assessed and found irrelevant.

At depth ten, RRF and BM25 recover the same judged-positive fraction in the first three results, but RRF places the first positive document later on average. At depths one and three, RRF matches BM25 on all three reported aggregate measures. Adding more candidates to the pool has not improved the final-cutoff recall in this sample. This is an observed result for a particular set of queries and depths, not a verdict on hybrid retrieval in general.

The aggregate loss is unusually concentrated. BM25 already ranks a judged-positive document first for ten of the twelve positive claims, leaving only claims 1368 and 873 with room to improve first-relevant rank. At depth ten, RRF demotes the positive document for claim 1337 from first to second. Its reciprocal rank falls from 1 to 1/2, and its binary nDCG@3 falls from 1 to approximately .6309. That one claim accounts for the RRF deficits in MRR@3 and nDCG@3 relative to BM25; the other positive claims have identical values on the three reported cutoff measures. No such differences appear at depths one or three. These conditions give us one observed ordering loss and little room to demonstrate a first-rank gain. They do not rule out gains in deeper recall or ranking multiple positive documents.

Claim 873 exposes a different failure. Five documents are annotated positive. Dense search places one of them third, giving Recall@3 of .2 and reciprocal rank 1/3. The depth-ten union contains two positives, but RRF selects neither in its first three. The evidence is available to fusion and absent from its selected prefix. Enlarging a candidate pool raises the possible recall ceiling; it does not guarantee that the final ordering reaches it.

The fictional section-based development collection shows the same separation under a different measure. At generator depth ten, the complete union covers all nine answerable source bundles, yet RRF’s first three cover only 7/9; BM25’s first three cover 8/9. These are authored bundle-coverage judgements, not SciFact document-recall results. In both settings, evidence in the pool can be lost at the ranking-and-selection boundary.

Agreement is useful evidence, with limits

Bruch and colleagues examine another way to combine lexical and semantic retrieval: normalise the two score sets, then take a weighted combination with weights summing to one. Unlike RRF, this approach retains information about score differences. But the choice of normalisation and weight becomes part of the method, and the combined values are not automatically calibrated probabilities. We have not implemented or tuned this alternative.

The published study also uses SciFact and the same named MiniLM checkpoint, but its evaluation is not equivalent to ours. It considers top-thousand generator lists, different BM25 parameters and SciFact nDCG at one hundred, and reports stronger nDCG from a tuned convex combination than from RRF under its conditions. Our depth-ten comparison, measured at three documents over twelve judged-positive claims, neither reproduces nor contradicts that outcome. List depth, ranking conventions, query sample and cutoff all affect what a result can support.

RRF rewards candidates found near the top by more than one retriever. That agreement can be useful, but the methods can agree on the same misleading topical neighbour. A precise identifier match from one generator may even lose to a weaker candidate supported by both. Neither agreement nor rank gives a document authority, establishes eligibility or shows that its evidence satisfies the task.

The pipeline keeps those responsibilities separate. Union rejects inconsistent views; fusion retains each generator’s score, rank and model provenance so the ranking can be inspected; final projection rechecks current eligibility before evidence is exposed. None of those recorded scores silently authorises a source or changes its permissions.

The experiment leaves us with two distinct options. We could change candidate generation so that different evidence enters the pool, or change the way the existing pool is ordered. Both need new, separately declared comparisons. Learned Sparse and Multi-Vector Retrieval first examines richer candidate representations and distinguishes executable mechanism demonstrations from models actually evaluated. The question of whether better pairwise scoring repairs the observed ordering losses comes after that.

References and further reading

Background and extensions for this chapter:

• Cormack, Clarke and Büttcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR. Defines the rank-based fusion mechanism; its title reports results for its own experiments.
• Bruch, Gai and Ingber (2022). An Analysis of Fusion Functions for Hybrid Retrieval. arXiv preprint, v1. Reports stronger nDCG for a tuned convex score combination than RRF under its tested conditions, and examines rank-fusion parameter choices.
• Luan et al. (2021). Sparse, Dense, and Attentional Representations for Text Retrieval. TACL. Investigates complementary sparse and dense representations behind hybrid retrieval.

Related chapters

• Retrieval Is a Policy. Places fusion inside an explicit retrieval policy and trace.