# Metrics that Answer Different Questions
# Book: Retrieval From First Principles
# Chapter URL: https://programmer.ie/books/retrieval/23-chapter/
# Prose fingerprint (sha256, normalized): ad5a2037cf8dea516f730542807d537886e6fa6859e7ec9e5cdc7586acd2c7aa
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

Suppose a retriever moves a relevant document from second place to first. Reciprocal rank improves, because it rewards finding the first relevant result sooner. But if the task requires that document and another document that was never retrieved, the evidence bundle remains incomplete. The ranking improved; the task’s evidence requirement did not.
Reciprocal rank has answered exactly the question it was designed to answer: how early does the first relevant item appear? The mistake is to treat that answer as a measurement of whether all necessary evidence was recovered.
Relevance Judgments and Test Collections established who judged the evidence, at what granularity and under which assumptions. This chapter uses that contract to ask separate, measurable questions about concentration, coverage, early ranking and graded relevance. No one metric substitutes for the others or establishes that the resulting evidence was used correctly.
Define the evaluated prefix For one information need, let R be the set of document IDs with a known positive relevance grade. Let the returned ranking be truncated to its first k distinct IDs; the external experiment fixes k = 4. Every formula below must be read with that cutoff and the corresponding judgement set in mind.
The evaluator rejects repeated result IDs: returning the same document twice cannot count as two discoveries. Search methods resolve score ties using the established full-reference order, which the evaluator preserves rather than breaking ties in favour of relevant documents. A document ID is unique within this external source and current view; when results span authorities, the complete source-qualified reference remains the identity that matters.
An unjudged document is not thereby irrelevant. For conventional point estimates, the evaluator treats unknown labels as zero, while separately reporting how much of the selected prefix actually received a judgement. This is a reproducible scoring convention applied to incomplete information, not a new negative judgement about the unassessed documents.
Count useful items and recovered evidence separately Precision at a cutoff asks how much of the allowed result space is occupied by known-relevant documents. It measures concentration within the prefix rather than how much of the full relevant set has been recovered:
P@k = number of known relevant results in the prefix / k Here the precision denominator is the requested cutoff k, even if the retriever returns fewer documents. At cutoff four, a single relevant result therefore gives P@4 = 1/4, not one. Other conventions exist, but all compared methods must use the same definition; otherwise a change in the metric masquerades as a change in retrieval quality.
Recall at a cutoff reverses the viewpoint: of all documents judged relevant for this need, how many appear among the selected results?
Recall@k = number of known relevant results in the prefix / |R| Suppose exactly two documents are known relevant. Returning one as the only result at cutoff one gives P@1 = 1 and Recall@1 = 1/2: the result is concentrated, but half the judged evidence is missing. If the first three results contain both positives with a known-negative document between them, P@3 = 2/3 and Recall@3 = 1. The larger prefix recovers more evidence while also occupying another result position and potentially more text.
Neither measure proves that the question can be answered. The external judgements identify relevant entities or documents. They do not specify whether both positives are jointly required, whether one suffices or whether a contradiction prevents accepting the evidence. Coverage of labels must not be confused with coverage of an evidence obligation.
Metric Question it answers Denominator What it cannot report Precision@k How much of the allowed prefix is known relevant? k, kept even when fewer results are returned Whether the prefix covers the required evidence Recall@k How much known relevant material survived into the prefix? |R|, the known-relevant set Whether every returned positive is needed together Reciprocal rank@k How early did the first relevant result appear? A position, averaged across needs Anything about further relevant items Average precision@k How concentrated are all known positives in the prefix? |R| Whether one grade is more useful than another nDCG@k How does the graded prefix compare with its ideal order? Ideal DCG@k for the same judgements Whether the evidence is true, sufficient or admitted Every row treats an unjudged return as zero under this evaluation convention, while companion fields record how much of the returned prefix was judged at all.
Book result: recall accounting audit. The same denominator problem appears outside ranked retrieval. A separate audit of Next’s research-candidate selector reconstructs fifty retained candidate IDs, their decisions and heuristic source/keyword-based labels. These labels are not independently adjudicated judgements of which discoveries actually proved useful. The table reports the recorded cohort, not a representative estimate of discovery quality:
Decision state Labelled positive Labelled negative Validated/promoted 21 0 Deferred frontier 6 1 Rejected 16 6 Whole cohort 43 7 With promotion as the endpoint, recall is 21/43 ≈ .4884: only twenty-one of the forty-three heuristic positives reached the promoted state. If the endpoint is instead retention for possible investigation, the six deferred positives also remain available, giving retention recall 27/43 ≈ .6279. That does not make those six validated discoveries. The corresponding promotion precision is 21/21 = 1; retention precision is 27/28 ≈ .9643. These are set-based decision measures, not ranked P@k, and their labels remain heuristic.
The selector’s own figure, 21/(21+16) ≈ .5676, leaves the deferred positives out of the denominator. It measures recall conditional on a decision having been made, not recall across all forty-three labelled positives in the cohort. Deferral need not be a poor operational choice, but excluding deferred cases changes the question being measured. A sound report names the endpoint, accounts for unresolved decisions separately, and computes precision and recall for the same endpoint.
The frozen calculation in research/calculations/next-evaluation-audit/ verifies the mutually exclusive decision groups and both recall conventions. The corresponding evidence-register entry identifies the inputs and limitations. This accounting check does not measure discovery recall beyond those fifty candidates, independently assessed candidate usefulness or any downstream benefit from promotion.
Measure where relevance appears Reciprocal rank ignores how many additional positives exist. It looks only for the first known-relevant item within the declared prefix:
RR@k = 1 / first relevant rank, or zero if none occurs by k MRR is the mean of these per-query reciprocal ranks over queries for which the metric is defined. It rewards moving the first positive closer to the top, not recovering every required document. In the opening example, moving the first relevant document from second to first changes RR from 1/2 to one while the required two-document bundle remains incomplete.
Average precision responds to more than the first positive: at every known-positive rank, it asks how much of the ranking up to that position is relevant. Our implementation stops at the declared cutoff:
AP@k = sum over positive ranks i <= k of P@i, divided by |R| The AP@k denominator is the number of all known positives, including those outside the returned prefix. Missing positives therefore reduce the score rather than disappearing from the calculation. This is explicitly truncated AP, not full average precision over a complete ranking. The IR textbook’s ranked-evaluation treatment supplies the background for precision at relevant positions and the distinction between ranked-prefix measures.
Preserve the meaning of grades Binary metrics treat every known-positive document alike. The external judgements also distinguish degrees of relevance, so we specify a gain function: 2**grade - 1 assigns gains zero, one and three to grades zero, one and two. At rank i, divide the gain by log2(i+1) to discount later results. Finally, compare the discounted total with the ideal ordering of those known grades at the same cutoff:
DCG@k = sum((2**grade_i - 1) / log2(i+1)) nDCG@k = DCG@k / ideal_DCG@k These are declared scoring choices, not properties guaranteed by the labels themselves. A highly relevant document moving earlier can increase nDCG while binary recall stays the same, because the grades and rank discounts contribute differently. Neither change verifies factual correctness, resolves a contradiction or validates an asserted relationship.
In particular, the chosen function gives grade two three times the undiscounted gain of grade one. The annotation scale is ordinal; it does not establish that this is the true ratio of user benefit. Järvelin and Kekäläinen developed cumulated gain, discounting and ideal normalisation, but implementations can choose different gains and discounts. A comparison that says only “nDCG” without those definitions is incomplete.
The implementation is small enough to run directly:
from retrieval_judgements import ranked_metrics ranked_metrics( ["B", "U", "A"], {"A": 2, "B": 1, "N": 0}, k=3, ) This is an executed numerical check, not a retrieved model prediction. A has grade two, B grade one and U is unjudged; the ranking is B, U, A. Under the declared unknown-as-zero point estimate, P@3 is 2/3, recall is one, reciprocal rank is one and AP@3 is 5/6. The observed DCG is 1 + 3/2 = 2.5; the ideal DCG is 3 + 1/log2(3), so nDCG is approximately .68853. Each value answers a different question about exactly the same ranking.
Only two of these three returned IDs have known labels. If U were also relevant, the precision at this cutoff could be as high as one; that is an optimistic bound, not the reported point estimate and not a new judgement. Reporting both the judged fraction and the bound exposes where incomplete assessment may change the interpretation.
Incomplete judging also motivates alternative measures. Buckley and Voorhees’ bpref compares known-relevant items with judged non-relevant items, rather than treating every unjudged result as an observed negative. That reduces one source of sensitivity to incomplete assessment, but it cannot create missing labels or establish whether the evidence answers a question; sparse positive judgements can make its preferences coarse. We report our declared conventional metrics and judged-fraction diagnostics. We have not computed bpref or inferred a corrected score from these traces.
Undefined does not mean zero A query with no known-positive documents has an undefined recall denominator. In our contract, AP and ideal-normalised nDCG are also undefined, and we leave reciprocal rank undefined because there is no established positive set against which to assess the first hit. The evaluator records nulls and reports the number of queries contributing to each aggregate. It does not silently turn missing judgements into zero-quality results.
Contrast that with a query that does have known positives but retrieves nothing. Its recall, RR, AP and nDCG are zero: the evaluated prefix missed the judged evidence. The judged fraction of an empty list is still undefined, because no returned item could have been judged. Tests keep these cases separate and reject invalid cutoffs, grades, duplicate IDs and mismatched labels.
A zero-positive qrels set is not proof that the collection lacks an answer. Some positives may simply be unjudged. Precision on such a query cannot establish that an abstention was correct. For our separately constructed absence controls, we instead rely on an explicit corpus-level absence judgement and record candidate returns and stop behaviour under that declared assumption, as explained in Chapter 22.
Cover the obligation, not merely a label A document metric checks labelled items; an evidence-bundle metric checks the parts the task requires together. Bundle coverage succeeds only when the selected units jointly contain every exact anchor in at least one accepted bundle. Single-unit sufficiency is stricter about packaging: one selected unit must contain all anchors. Compatible separate units may cover a bundle together, but overlapping only part of a required source span is not enough.
The partial-absence control names an exact anchor for the reset instruction that does exist. It also requires another procedure absent from the declared inventory. Retrieving the reset passage therefore covers one component without satisfying the complete need. We do not manufacture a quotation or source span for the missing procedure to turn partial evidence into a positive bundle.
A second executed control deliberately assembles three source clauses for inspection: a positive project assignment, its positive contact relation and a current opposing assignment. The positive anchors are all present, so structural bundle coverage succeeds. The source-backed path validator nevertheless refuses the relationship because the opposing assignment remains unresolved under its declared conflict rule. These clauses form an inventory for the metric control, not a packet that the retriever actually returned. This isolates the difference between covering positive text and accepting a valid relationship.
A candidate pool may contain every required document, giving candidate recall one, while the final selected packet omits one of them. With two required documents, selecting only one gives selected recall one half and leaves the bundle incomplete. The executed hand control deliberately allows one selected unit to expose that distinction; the external experiment’s four-unit allowance must not be confused with this control.
A relevance label or high ranking score cannot validate a fabricated source reference, restore a revoked document version or make restricted evidence eligible. Reference integrity and current-view eligibility remain separate checks before a selected result may be exposed.
Score the decision without making it the answer A retrieval controller can follow every execution rule and still return weak evidence. Stopping at an exhausted budget is a truthful account of why execution ended; stopping because a term-overlap proxy claims sufficiency is an inference that may be wrong. Neither case is a demonstrated error in a generated answer, because no answer generation was evaluated. Control-flow correctness and evidence quality must be reported separately.
The external results expose that separation numerically. Adaptive MRR@4 is .7500, slightly higher than lexical search’s .7451, yet adaptive mean recall is lower, approximately .1622 against .1699. It gets the first judged-positive result early a little more successfully while recovering fewer judged positives overall. Both numbers concern document-level qrels; neither is a measurement of complete evidence bundles or generated answers.
The executable examples, edge-case tests and separate arithmetic audit recalculate these quantities from retained identities, grades and exact spans. They verify the calculations under their stated judgement assumptions; they cannot certify those assumptions or prove a retrieval policy caused an improvement. Controlled Experiments and Regression Gates takes up that separate question: what does the experimental comparison itself justify us in concluding?
References and further reading Background and extensions for this chapter:
Järvelin and Kekäläinen (2002). Cumulated Gain-based Evaluation of IR Techniques. ACM Transactions on Information Systems. Provides the graded-gain and rank-discounting foundation for DCG-style evaluation. Buckley and Voorhees (2004). Retrieval Evaluation with Incomplete Information. SIGIR. Extends metric choice to incomplete assessment through bpref. Voorhees (2000). Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. Information Processing & Management. Connects calculated system comparisons to variation in the underlying relevance judgements. Related chapters How Do You Evaluate an Embedding?. Distinguishes representation metrics from application-level evaluation.