Diversity, Coverage, and Evidence Budgets

For SciFact claim 1337, the annotated positive document survives candidate generation and ranks second after cross-encoder reranking. Yet it never reaches the evidence packet. Its title-and-abstract text contains 1,298 characters, more than the entire 1,200-character allowance, so the selector skips it and returns a different, 1,108-character document. Moving the positive document to first place would not help: as long as selection admits only whole units, that document cannot fit. The loss occurs at the budget boundary, not because the ranker failed to retrieve it.

Rerankers and Cross-Encoders improved some fictional development rankings without improving the external aggregates. But an ordered list is not yet an evidence packet. This chapter asks which complete units survive a finite content allowance, how much judged evidence is lost in making that selection, and whether discouraging redundancy helps under the same constraint.

State the resource before comparing methods

Part II compared systems that returned three units per query. Three short blocks, however, can contain far less text than three long sections. A common result count does not impose a common burden on the next system. We now constrain the amount of selected evidence text explicitly, rather than treating the number of results as its cost.

This does not repair the missing equal-budget comparison from Part II. We have not tested blocks, windows and sections under equal character allowances or under the actual token budget of a consumer model. The present experiment selects existing whole-document SciFact candidates. It can show what this allowance excludes, but cannot determine which chunking scheme would be preferable under an equivalent content budget. That comparison remains not measured.

Every selector receives the same depth-ten exact-reference union, already ordered by the same cross-encoder. The document judgements and candidate identities do not change between selectors. Only the rule for converting that order into a selected packet changes:

The allowance is measured in Unicode codepoints of the exact raw unit texts, with two LF characters charged between selected units. Selection keeps whole units: it neither clips an oversized document nor creates a shorter substitute. When a unit will not fit, the character-based selector skips it and continues down the order in search of a feasible one. An empty packet costs zero characters. Filling every available character is not an objective, and a partial prefix is not accepted as the original evidence unit.

This is a budget for the selected evidence body, not for a complete model request. Citation formatting, surrounding instructions and other consumer context are outside the cap. A real model-token allowance would have to count the final serialised packet with that model’s tokeniser. The regex-based word counts retained here are diagnostics only; they do not justify converting 1,200 characters into a promised number of model tokens.

The selectors and the 1,200-character cap were fixed before evaluating the final sample. Because the candidates are whole title-and-abstract units, this is a stringent feasibility test. We do not change the budget, split documents or shorten representations after seeing which positive documents fail to fit. The inconvenient exclusions are part of the result.

Separate redundancy from relevance

Even when several units fit, simply taking the highest-ranked ones can waste a packet on repetition. Two passages from one parent may overlap; different sources may restate the same claim; several top-ranked units may cover one aspect and omit another. Three returned results are therefore not necessarily three distinct evidential contributions. Relevance determines why a candidate is worth considering; diversity asks what it adds to what has already been selected.

Carbonell and Goldstein’s MMR work motivates trading query relevance against similarity to already selected material. Our implementation uses a specific greedy version, not an unspecified default MMR scorer. For each candidate that still fits the budget, with original cross-encoder rank r, it calculates:

marginal = 0.7 / r - 0.3 * redundancy
redundancy = max(0, cosine(candidate, each already selected unit))

Before anything has been selected, the redundancy penalty is zero. Afterwards, it is the largest non-negative cosine similarity between the candidate’s retained normalised dense vector and any already selected unit. The relevance term is based on the candidate’s original cross-encoder rank, not its raw logit or a probability; removing other candidates never renumbers that rank. Exact complete-reference identity breaks ties, so repeated executions need not depend on iteration order.

The algorithm scores only candidates whose entire text, including any required separator, fits the remaining allowance. A candidate cannot buy its way past the hard cap with a high relevance or diversity score. The .7 relevance weight and .3 redundancy weight are declared constants, not values optimised on the final sample. This makes the selection rule reproducible without asserting that the chosen trade-off is optimal.

A small hand-controlled case shows that the implemented penalty can favour a complementary vector over a more redundant one. That verifies the greedy mechanism in a case where it can actually make a choice. It does not establish that the alternative passage adds the information a real task needs; that question requires source-grounded judgements and a feasible multi-unit comparison.

Observe the price of the cap

The external whole-document results are severe:

The mean text and unit counts include all twenty queries, including those without judged-positive documents. Recall, reciprocal rank and nDCG average only the twelve claims with annotated positives; the other eight have undefined relevance values under this protocol. Under either constrained selector, nineteen queries receive exactly one whole unit and one receives none. More importantly, the two selectors produce the same ordered references on every external query. Their identical scores reflect identical packets, not an observed equivalence between relevance-only and diversity-aware selection.

The length distribution explains why the diversity mechanism scarcely comes into play. Among 5,183 unique SciFact documents, the median title-and-abstract unit contains 1,429 characters—already above the 1,200-character cap. The sixteen distinct annotated-positive documents have a median of 1,736 characters; just two fit individually. Only three of the twenty depth-ten query pools contain any pair of distinct documents that could fit together, including the two-character separator. These figures are derived from retained lengths and pool membership, not from another retrieval experiment.

In those three potentially pair-feasible pools, both selectors still choose the same first feasible document, leaving insufficient space for a second. MMR’s redundancy penalty is zero at the first decision, so almost none of the external selections tests its defining trade-off. Its identical result to character packing is therefore a whole-unit feasibility limitation, not evidence that diversity has no value. Under this particular budget, there is effectively no second choice for the penalty to influence.

The unbounded count baseline answers a deliberately different question: what judged evidence survives when we allow three ranked whole units regardless of their total length? Comparing it with a capped selector exposes the cost of the restriction, not the superiority of one selection policy at equal content cost. Character packing and MMR do share the same cap, but identical mostly one-unit packets make their diversity comparison inconclusive. A future test would need a predeclared allowance that admits meaningful alternatives; neither a larger budget nor different retrieval units have been evaluated here.

Two claims illustrate different points of failure. For claim 1368, the annotated positive is missing from the candidate pool and none of the available candidates fits, leaving an empty packet. For claim 1337, the annotated positive is in the pool but exceeds the allowance on its own. Its loss occurs during whole-unit selection. Both bounded packets have zero recall, but their traces prescribe different investigations: candidate generation in one case, representational granularity or budget feasibility in the other.

Keep the teaching collection separate

The fictional section-based development collection gives a different view of the budget constraint. Across nine answerable needs, unrestricted count selection covers 9/9 bundles with a mean packet length of 1,055.4 characters over all ten queries. Character packing covers 8/9 at 384.9 characters, and MMR covers 8/9 at 413.6. Every method returns three units per query, yet the character totals differ sharply. These are source-bundle coverage figures, not SciFact document-recall measurements.

The development loss is concentrated in the long-selector need s-d06. Unrestricted count selection includes the required evidence in a 2,438-character packet; both budgeted methods return just 226 characters and miss the declared source bundle. The structural judgement records whether selected source spans meet the authored obligation. It says nothing about whether an answer-generating model would notice, trust or correctly use those spans.

MMR changes the selected order or membership for six of ten fictional development queries, but the aggregate bundle coverage remains 8/9. That shows the rule can change a packet when the units and cap permit it; it does not show an improvement in declared evidence sufficiency. A visibly different selection is an intermediate behaviour, not a downstream success metric.

What diversity does not decide

Low vector similarity does not guarantee complementary facts. Two differently phrased passages may make the same claim, while two highly similar passages may offer valuable corroboration—or a direct contradiction. A cosine-based redundancy penalty cannot determine whether a contradiction should be included, which source governs, or what set of claims a task requires. Diversity in representation space and completeness of evidence are different properties.

The selector reports the number of source namespaces and parent documents separately from the number of retrieval units. Several sections from one parent are multiple units but not multiple parent documents. In the SciFact adapter, every document belongs to the same dataset source namespace; distinct document IDs cannot be counted as independently verified authorities on that basis. These counts help reveal concentration, not source reliability or evidential independence.

Units with identical words retain their distinct exact references. A later duplicate-aware selector might group related text, but would still need to preserve attribution and state when omitting an occurrence changes the evidence requirement. The current MMR rule does not impose a source quota, require named aspects to be covered, resolve contradictory claims or optimise the selected set globally. Those are additional policies, not consequences of a cosine penalty.

After selection, current-view projection can still remove a stale, restricted or invalid reference, producing a generic empty packet when nothing remains. That protects this declared retrieval boundary; it does not prove that the permitted corpus lacks an answer. The fictional battery request receives candidates under every selector. All eight external claims without judged positives also receive bounded candidates, but incomplete positive annotations do not establish a false-answer rate. Deciding when to abstain requires a separately specified and evaluated rule.

Carry the remaining obligations forward

xQuAD offers a different way to think about diversity: model possible query aspects, then favour documents that cover important aspects not yet represented. Its guiding question is closer to “what obligation remains uncovered?” than MMR’s “how similar is this candidate to what I selected?” The approach needs query–aspect and aspect–document estimates absent from our current pipeline. Even explicit aspects would require validation against the task’s real source-evidence bundles. No aspect-aware selector or alternative content cap has been evaluated here.

Across Part III, the traces locate several different losses: judged evidence never enters the candidate pool; useful candidates fall below the final cutoff after fusion or reranking; model inputs omit source text through truncation; and whole units fail the evidence-packet budget. These failures happen at different boundaries, so one more sophisticated ranker cannot be presumed to repair them all. Candidate breadth controls an upper bound, ordering affects the prefix, and feasibility governs what survives selection. Admission, actual use and downstream helpfulness remain not measured.

The next question moves upstream: can the request tell us which route, vocabulary or subquestion is worth pursuing before spending the retrieval budget? Understanding Queries and Routing introduces a new constructed design to examine those choices. It preserves exact evidence identity and charges additional searches against explicit limits, but inherits the need for independent evaluation. A controller is useful only if we can inspect the evidence it obtains, omits, selects and leaves untested—not merely because it performs more steps.

Capstone checkpoint: inspect the pool and the packet

python -B -m examples.capstone.milestones 3 --summary

The capstone uses the existing lexical and dense generators, unions their exact references, runs RRF and invokes the library’s MMR selector under a declared character allowance. The summary exposes generator counts, leading fusion candidates and origin ranks, the selected order and the actual evidence-text size. The complete record is .artifacts/capstone/runtime/milestones/03.json; omitting --summary retains the original machine-readable JSON status. This demonstrates the selection mechanism, not an additional Pi search strategy or a measured gain in task usefulness. The whole-document external comparison remains inconclusive about diversity.

References and further reading

Background and extensions for this chapter:

• Carbonell and Goldstein (1998). The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. SIGIR. Provides the relevance–redundancy trade-off behind diversified selection.
• Santos et al. (2010). Explicit Search Result Diversification through Sub-Queries. ECIR. Extends diversity towards explicit coverage of possible query aspects.
• Liu et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. TACL. Motivates checking what a consumer can use after selection and ordering, beyond merely fitting text.

Related chapters

• Assemble for the Task. Continues from ranked candidates to feasibility and task-specific context assembly.