Filters, Scope, and Freshness

The index still ranks the handheld reset procedure first. But since that index was built, the document has become restricted. Removing the hit from the public results prevents that particular disclosure. If filtering happens only after a short ranked list is returned, however, it may also hide permitted evidence farther down the list.

Exact Search, ANN, and Index Choice asked whether approximate search preserved the neighbours found by exact search. This chapter adds a different test: even when an index retrieves the intended passage, is that passage still permitted for this request? We must examine both the policy decision and the point in the search process where it takes effect.

Separate an indexed observation from a current decision

The collection uses named views that declare source scope, current version, lifecycle and freshness. Those declarations are treated as trusted inputs. The caller chooses a view, but this teaching implementation neither authenticates that caller nor discovers which policy state is actually the latest. Everything that follows depends on that boundary.

The Part II fixture introduces five later views of the same scoped-reset document: restricted, stale, revised, removed and unknown scope. Each view changes only that document’s status or version. The revised view points to a retained twenty-second instruction, while the old index still holds the ten-second version. Other documents remain eligible, so we can see whether filtering excludes the outdated evidence without needlessly losing permitted alternatives.

Later view	What it changes for this document	What the wrapper does with old indexed rows
Restricted	Scope no longer permits public disclosure	Refuses the old rows; permitted alternatives can still fill the budget under pre-filtering
Stale	Freshness no longer supports current retrieval	Same refusal; staleness is not a claim that the words are false
Revised	A newer version exists, with new unit identities	Refuses the old rows; a refreshed manifest rebuilds the changed document
Removed	The document leaves the current collection	Refuses the old rows; permitted historical resolution remains separate
Unknown scope	Scope cannot be established	Fails closed and excludes the rows

When building a fresh index, build_units first selects the source versions permitted by the declared view, then parses and encodes them. Excluded material therefore does not become a unit or vector in that newly built view. But an index created under an earlier, more permissive view still contains its old entries. The offline DenseIndex.rank backend has no independent access-control boundary; wrapping or labelling its output as public cannot make the backend a protected service.

The public_search wrapper provides the additional gate. On every call, it derives a permitted set of source, version and content-digest identities from the caller-supplied trusted view, then intersects that set with the indexed units before exposing evidence packets. Unknown scope is excluded. An old passage cannot become current merely because its text digest matches another record: its source and version must agree too. This recheck enforces the supplied view, not an independently verified latest policy.

A returned packet still describes the observation made when its index was built. Separately, the outer policy_view identifies the view used for the latest eligibility check. Keeping both facts visible prevents an old indexed observation from masquerading as freshly collected evidence. A production design might separate these fields differently, but it cannot safely treat index time and policy-check time as the same event.

Filtering position changes the search problem

The placement of the eligibility check changes which candidates can fill the result budget. With pre-filtering, the exact scorer limits its search to currently permitted rows and then takes the requested top results. With bounded post-filtering, the backend first chooses a fixed-depth list without that restriction; the wrapper removes forbidden rows afterwards. Unless it searches again, a discarded top hit leaves a gap that a lower-ranked permitted hit cannot fill.

Consider the hand-controlled query vector [1,0]. The closest record has vector [1,0] but is excluded; a permitted record has vector [0.9,0.1] and ranks just behind it. We request one result and set the post-filter depth to one. Pre-filtering excludes the forbidden record before ranking and returns the permitted alternative. Post-filtering receives only the forbidden top result, removes it, and returns no_candidates. The permitted record was present but never reached that one-item post-filter list.

These vectors are a deliberately constructed control, not MiniLM outputs. They isolate a filtering-starvation failure: the search procedure returns no public candidate even though the eligible indexed set contains another row. no_candidates describes what survived this procedure. It does not mean the permitted collection lacks potentially useful information, and the control does not establish that the alternative would answer the user’s task.

Fetching a deeper initial list can reduce starvation in some cases, but no fixed overfetch depth guarantees a permitted result under every exclusion pattern. Repeating the search, increasing ANN probes and filtering during index traversal each change the algorithm and its cost; none was evaluated as a general remedy here. Our approximate adapter can rank allowed rows already present in its candidate pool, but it cannot retrieve an allowed row that approximation omitted from that pool.

Run the stale-index case with learned vectors

A graph-based ANN index introduces an additional difficulty: removing disallowed nodes can disrupt paths used to navigate towards allowed neighbours. ACORN addresses predicate filtering by searching the predicate-induced subgraph, building a denser graph to help preserve navigability and falling back to pre-filtering for very selective predicates. Selectivity and how allowed nodes sit within vector neighbourhoods both matter. Our exact pre-filter path simply scores allowed rows; it implements none of ACORN’s graph traversal. In either design, a predicate match is only a filtering result, not proof that the metadata is trustworthy or the requester is authorised.

The current-view gate comparison now applies the already encoded scope query to the unchanged baseline dense index. Every later view makes the indexed old scoped-reset units ineligible. Exact pre-filtering returns three permitted candidates; post-filtering at depth one returns none. The difference is caused by when the current-view gate acts, not by a changed query embedding or a newly learned ranking.

Indexed scheme	Baseline indexed rows	Rows still allowed from old index	Refreshed rows after revision
Blocks	33	28	33
Windows	67	63	67
Sections	17	15	17

The table separates three quantities: rows in the original index, rows from that index still permitted by the later view, and rows after rebuilding for the revision. The old index cannot provide the newer version’s text or vectors. A fresh revision-view manifest restores the original row counts, but with different identities for the changed document. The capture rebuilds unit manifests and lexical statistics; equal row counts are not evidence of equal content, and improved dense task quality was not measured.

Across all five later views and all three chunking schemes, the same contrast appears: exact pre-filtering refills from permitted rows, while depth-one post-filtering returns nothing. Public results contain no excluded text or identities. The retained fixture and policy audit hold more detailed offline inventories for verification, not for public disclosure. The wrapper reports allowed-row counts but does not reveal which protected source caused an exclusion.

One negative control changes an excluded row’s vector and checks that the safe exact public response does not change. Other tests cover empty eligible sets, malformed vectors, refusal of old references and unknown scope. These checks support the behaviour of the wrapper under trusted local fixtures; they do not test an adversary’s ability to forge a policy view, bypass the wrapper or exploit a deployed service.

Refresh statistics as well as visible rows

Filtering affects more than which rows appear in a result list. BM25 scores depend on the collection used to calculate term frequencies, document frequencies and average passage length. Hiding restricted hits while continuing to use statistics derived from them is a different scoring policy from building those statistics over the permitted view. Here we rebuild the lexical index from refreshed eligible units using the original implementation.

The recorded section-level view makes that difference concrete. Restricting the scoped-reset document leaves fifteen indexed rows with mean token length about 32.867. Rebuilding for the revised view produces seventeen rows with mean length about 30.941. These are properties of the constructed collection, not improvements in retrieval quality. Because the corpus statistics differ, scores from the two views are not automatically comparable even for an unchanged query.

Dense cosine similarity does not use BM25’s document-frequency statistics, but cached vectors still have dependencies that must be tracked. Revising a source changes its text and unit identities; changing the encoder configuration changes its model fingerprint; changing policy changes which indexed rows may be exposed. A cache keyed only by document name would hide those distinctions and could serve an obsolete representation or an ineligible reference.

This is an instance of Saltzer and Schroeder’s complete mediation principle: remembered access decisions must remain valid when authority changes. Vector freshness and permission freshness are different obligations. Rebuilding an index, deleting a posting or checking one reference does not, by itself, prove that every cached public result reflects the currently applicable access decision.

Historical evidence does not grant current access

Retaining an old version allows us to identify what a historical reference originally named; it does not authorise showing that text today. The new manifest resolver checks eligibility under the supplied current view before returning the exact slice. In every later view, including the revision, old scoped-reset references return the generic status unavailable. They are never silently redirected from the old ten-second instruction to the newer twenty-second one.

There is an important limitation in the earlier historical resolver. A trusted caller can still select a permissive old view and retrieve text that a later view restricts. That resolver does not discover or enforce the latest policy. The new wrapper expects a trusted current view, but authenticating the caller and determining which view is authoritative remain outside this experiment. Neither interface, as demonstrated, establishes production-grade access control.

Keep retrieved words in the data boundary

A fictional imported note also contains hostile instructions. For a legitimate request about the note, retrieval can return it with its ordinary source identity. The text remains evidence data: parsing, embedding and ranking do not execute it or allow it to override the selected policy view. We measured retrieval of the note, not how an answer-generating consumer would respond to it. Prompt-injection resistance downstream is therefore not measured.

The seven-stage chain remains available → retrieved → eligible → selected → admitted → used → helpful. This chapter strengthens the eligibility boundary for the declared view, but it does not determine whether any surviving candidate is relevant or sufficient. Nor does it measure what a later consumer admits, relies on or finds helpful. Even the permitted record recovered by pre-filtering in the constructed control is not automatically a correct answer.

We have now distinguished two reasons for an apparently missing result: the index may fail to find a candidate, or the filtering procedure may discard it before a permitted alternative can fill the budget. A third problem remains even when retrieval and eligibility both succeed: the passage may be similar to the request yet misleading, contradictory or incomplete. Semantic Failure Analysis and Hard Negatives examines those failures before we add another retrieval mechanism.

References and further reading

Background and extensions for this chapter:

• Patel et al. (2024). ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data. arXiv preprint, v1. Studies vector search combined with structured predicates; predicate matching does not establish permission.
• Singh et al. (2021). FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search. arXiv preprint, v1. Extends freshness handling to index insertions and deletions.
• Saltzer and Schroeder (1975). The Protection of Information in Computer Systems. Proceedings of the IEEE. Provides architectural background for rechecking access and refusing unauthorised disclosure.

Related chapters

• Context Goes Stale. Separates elapsed time from changes that actually make information stale.
• Keep Context in the Right World. Extends scope checks to the world and project to which context belongs.