← Retrieval From First Principles

Relevance Judgments and Test Collections

Define information needs, preserve judgement provenance and keep unknown relevance distinct from absent evidence.

A document explicitly states that no battery-replacement procedure is recorded. Should a retriever mark it relevant? The answer depends on what the user actually needs.

If the user asks whether the notes contain a procedure, that negative statement may be useful evidence. If the user asks how to replace the battery, it does not supply the requested steps. The text is identical in both cases; only the information need changes. Relevance is therefore a relationship between a need and evidence, not a permanent property of a document.

Bounded Adaptive Retrieval: When to Stop made decisions about continuing search from what the controller observed. To evaluate those decisions, we need an account of relevant evidence that exists independently of the controller’s observations and choices. Otherwise its own notion of sufficiency could become both the decision rule and the marking scheme. A controller’s confidence cannot write its own relevance labels.

Judge a need, not just a string

An information need describes what the user wants to know or accomplish. A query is the particular wording sent to a search system. A judgement relates that need to a specified evidence unit under stated assumptions. Keeping all three identities separate matters: two queries may express one need, while two superficially similar queries may ask for different evidence. Neither shared words nor shared query results justify copying the labels.

The retrieval unit also determines what a positive judgement means. A document-level label indicates relevance under a document-assessment protocol, but does not locate a supporting sentence. A passage-level label needs an exact span and qualified source. A sufficient-evidence bundle requires several specified pieces to be present together. A relation-path judgement additionally requires a valid connection between statements, not simply their co-occurrence.

Judgement unit What a positive records What it leaves open
Document The document is relevant under a document-assessment protocol Which sentence supports the need, and whether the document alone suffices
Passage One exact span in a qualified source supports the need Whether the complete required evidence is present
Sufficient-evidence bundle Several specified pieces are present together Whether the text is true, current under another view, or usable by a consumer
Relation path A valid connection holds between statements Whether the supporting sources are authoritative, or whether another source contradicts them

A third state sits beside these: an unassessed item is unknown, not judged non-relevant. The convention that scores unjudged returns as zero is a scoring choice, not an assessment record.

These are distinct evaluation questions, not interchangeable versions of one positive label. An article may be relevant yet insufficient to answer a request on its own. Even when every statement in a proposed chain is found, an invalid or contradicted join may prevent the chain from satisfying the need. Choosing a metric before choosing the judgement unit would conceal these differences.

The IR textbook’s discussion of relevance assessment explains why information needs, assessment and pooling belong in the same evaluation design. In conventional pooling, several retrievers contribute results for human assessment, making a manageable set rather than judging every item in a large collection. That economy creates an important unknown: an unassessed item is not thereby irrelevant. This is methodological background, not verified provenance for the particular collection examined next.

Use external judgements without giving them extra authority

The new external evaluation uses NanoDBPedia’s 6,045 entity abstracts and fifty queries. Its relevance judgements trace back to DBpedia-Entity v2, whose entity-oriented information needs came from several evaluation campaigns. Crowd assessments, with expert follow-up, assigned grades for irrelevant, relevant and highly relevant entities. That external provenance is valuable because the book’s implementing author did not invent the positive labels for this collection; it does not make those labels exhaustive or infallible.

The primary paper’s §2.3.1 describes how the upstream assessment pool was assembled: previously relevant entities, results from older runs mapped to the newer DBpedia version, new retrieval runs and SPARQL results for QALD queries were brought together for rejudgement. This tells us where the upstream candidates for assessment came from. It does not mean every entity was assessed, and it does not explain how NanoDBPedia selected its smaller set of queries and documents. Those are separate provenance questions.

NanoDBPedia’s pair file identifies positive query–entity pairs but omits their original graded values. The preparation step checks all 1,158 pairs against the primary positive labels before restoring grades zero, one and two; it also retains available explicitly judged zeros. This is a verified transformation of inherited assessments, not fresh labelling or a second reviewer. The inspected Nano card does not describe its query or document selection procedure. Consequently, we preserve the recorded counts and identities while leaving representativeness and sampling bias unresolved; we do not call the subset random or unbiased.

An upstream agreement file is retained as part of that provenance. It does not let us reconstruct unrecorded individual votes or erase assessor disagreement. External authorship improves the independence of these judgements from our retrieval code, but public availability, incomplete coverage and fallible assessments remain separate limitations.

Retrieval preserves the original entity identifiers and complete source/version references, so a returned abstract remains traceable to the assessed record. But a historical judgement that an abstract was relevant does not certify current facts about the person or organisation it describes. Likewise, the wrapper’s “current” view is current only relative to its frozen fixture, not the live Web. If publication time is unknown, it stays unknown; fixture timestamps must not be recast as observed acquisition events.

Separate four experimental roles

Before inspecting retrieval outcomes, we assigned the fifty external query IDs to experimental roles using a fixed hash ordering. The division matters because debugging a tool, choosing a stopping threshold and evaluating the selected threshold are different activities:

Role Queries Permitted use
Development 8 Check interfaces and inspect feasible budgets
Policy selection 8 Choose between the two declared thresholds
Final evaluation 34 Evaluate the frozen selected policy
Mechanism validation Separate constructed requests Check exact bundles, paths, absence and permissions

The thirty-four final questions stayed fixed when a baseline or the controller performed poorly. Neither their examples nor their retrieval outcomes were used to design the controller or choose its stopping threshold. This protects the declared comparison from post-result replacement. It does not make the evaluation blind: the source data and primary query file were public and accessible during source verification.

The external roles share no query IDs or exact query formulations, but they do share a document collection. Nine known-positive documents occur in both development and final. That is normal for a query-based retrieval comparison, yet it limits the novelty of the content being searched. This is a new evaluation collection for the book, not a full benchmark, a test on wholly unfamiliar documents or an audit establishing that model training never encountered the underlying material.

The pinned dense model card lists Simple Wikipedia, Natural Questions and TriviaQA as training sources. Because DBpedia abstracts derive from Wikipedia, overlapping underlying content is plausible. We have not audited exact exposure to the selected entities or documents, how much overlap occurred or whether it helped or hurt retrieval. The result is an explicit uncertainty, not a licence to call the evaluation contamination-free or to claim proven contamination.

The constructed mechanism controls serve a different purpose from the external final questions. Twelve reuse Part IV development requests, and five add controlled absence or view-change challenges. Their author-written labels make expected behaviours inspectable, including cases whose complete fictional inventory is known. Putting them in a new directory does not make their evidence independent. Nor do we reclassify Part IV’s mirrored final world as a previously unseen test.

Make unknown a supported state

An omitted relevance judgement is unknown, not a judgement of irrelevance. The NIST qrels explanation distinguishes documents that have not been assessed from explicitly judged nonrelevant ones, even though conventional evaluation often scores unjudged returns as if they were nonrelevant. We can adopt that scoring convention for a reproducible comparison without changing the underlying assessment record into a zero. The distinction matters whenever a retriever surfaces evidence outside the original pool.

Our judgement schema records the need, query, experimental role and granularity together with an exact evidence reference, grade, judgement source, annotator provenance, uncertainty, relation type and eligibility assumptions. The grade can explicitly be unknown. Where an illustrative assessor disagreement is unresolved, the alternative assessments can remain visible without manufacturing a resolved grade. Otherwise an apparently precise label could conceal the very uncertainty the evaluation needs to preserve.

examples/retrieval_judgements.py validates this contract before any score is calculated. It refuses contradictory duplicate labels for the same need and evidence reference, unresolved references, passage quotations that do not match the source, incompatible granularity and judgements attached to the wrong experimental role. It does not reject an item for having different labels under different information needs: that is exactly the distinction established by the opening battery example.

A passage judgement must point to a span that reconstructs the retained source text exactly. Splitting or merging chunks changes retrieval-unit identities, so a document label cannot simply be reassigned to a new chunk ID. Source qualification matters as well: matching words from another publisher, version or permission scope do not automatically inherit the original judgement. Re-evaluation needs an explicit mapping from the assessed evidence to the new unit.

Absence needs a corpus-level assertion

A query with no positive qrels does not necessarily describe a question the collection cannot answer. Relevant evidence might never have entered the annotation pool, might remain unjudged, or might be labelled at a different granularity. An empty positive set is a fact about the available judgements; a claim that no sufficient evidence exists is a separate assertion about the corpus. Search failure establishes neither.

For the external entity questions, answerability therefore remains unknown. Known-positive documents tell us something about annotated relevance, not whether a particular selected packet provides a complete answer. The constructed battery example is different: the author has declared and inspected a complete fictional inventory containing no requested procedure. Its absence judgement applies only within that specified inventory and need, not to the world outside the fixture.

The partial-absence control records an exact anchor for the reset instruction that exists and a separate obligation for a procedure missing from the declared inventory. A selector may recover the available anchor without satisfying the combined request. Giving credit for the whole need would confuse partial coverage with sufficiency; inventing a source span for the missing procedure would be worse, because it would fabricate evidence.

A final complication is permission. Evidence can exist in a historical version yet be unavailable to the caller in the current view. An authorised offline test may distinguish exclusion, genuine absence and retrieval failure, while the public interface must not expose protected source or relation identities to explain its refusal. A generic unresolved-path response can preserve that boundary without pretending all internal failure causes are identical.

Keep the grader outside the decision loop

The controller accepts the request and eligible view, not the target evidence or relevance labels. Evaluation loads those labels only after retrieval has produced its results. The separate policy-selection stage may inspect scores on its designated slice, but final evaluation does not trigger another round of threshold changes. This keeps the intended decision path from using the answer key to decide whether to search again.

That separation is enforced by the implemented interface, tests and frozen procedure. It is not adversarial isolation: arbitrary replacement Python code in the same environment could still read files. Our claim is that the evaluated controller did not receive judgement fields through its declared request contract, not that the process is technically incapable of accessing them.

The remaining uncertainties affect how any aggregate should be read. NanoDBPedia’s subset-selection procedure is undocumented in the inspected source; the primary work describes upstream pooling, and unjudged entities remain possible. Moreover, a four-document packet cannot contain every positive entity for a broad list request. These are limitations of the question set, annotation coverage and output budget. They belong beside the findings, not as reasons to remove difficult queries from the comparison.

The evaluation contract now identifies the information need, assessed unit, label provenance, unknowns and experimental role. A metric is meaningful only when it respects those definitions. Metrics that Answer Different Questions turns the labels into reproducible calculations without silently converting unjudged evidence into known negatives or partial coverage into a complete answer.

References and further reading

Background and extensions for this chapter: