Breaking the Detector

Chapter 6 changed the standard for evaluation.

A detector is not trustworthy because it has a good AUC.

It becomes trustworthy only after we understand how its errors behave under the conditions in which policy will use it.

That immediately suggests a more aggressive question.

Instead of waiting for those conditions to appear naturally, can we search for them deliberately?

Suppose a containment detector works well on ordinary unsupported claims.

Do not celebrate yet.

Ask:

What is the smallest truth-bearing change that the detector fails to notice?

What wrong evidence looks safest to this exact detector?

What valid evidence can be removed before the score should become suspicious?

What happens when the attacker sees the score and searches against it?

What configuration change moves the decision boundary without changing the underlying claim?

Which upstream verifier stage can be corrupted most cheaply?

Ordinary evaluation asks:

How does the detector behave on examples we have?

Adversarial evaluation asks:

Can we construct valid examples specifically designed to exploit what the detector does not observe?

Hallucination Energy will remain the running victim because we know its mathematics, implementation and initial failure modes.

But the method is broader.

Every hallucination detector has an observation channel.

Every observation channel preserves some distinctions and discards others.

The red-team job is to find a failure that lives primarily in the discarded information.

A good adversarial example does not merely make the detector wrong. It explains why the detector could not see the failure.

That turns red teaming into more than attack generation.

It becomes experimental falsification for measurement systems.

1. Start with a threat model, not merely an attack idea

Chapter 4 gave us a measurement contract.

Chapter 6 gave us an evaluation contract.

Adversarial testing needs an attack contract that also functions as a threat model.

A useful version looks like this:

attack_contract = {
    "victim": {
        "detector": "hallucination_energy",
        "detector_version": None,
        "embedder_version": None,
        "rank": None,
        "segmentation": None,
        "threshold": None,
        "calibration_version": None,
    },
    "objective": {
        "target_failure": "unsupported_claim",
        "attack_goal": "false_acceptance",
        "oracle": "claim_evidence_support_label",
    },
    "attacker": {
        "knowledge": "score_access",
        "capabilities": ["edit_claim", "choose_evidence_set"],
        "query_budget": 16,
        "text_edit_budget": None,
        "evidence_addition_budget": None,
    },
    "validity_constraints": [
        "oracle_label_preserved_or_changed_as_declared",
        "reference_remains_fixed_unless_evidence_is_attack_surface",
        "example_remains_well_formed",
    ],
    "evaluation": {
        "victim_frozen_before_attack": True,
        "threshold_frozen_before_attack": True,
        "held_out_attack_set": True,
    },
}

Three attacker properties should remain separate.

Knowledge

What can the attacker observe?

decision only
score
algorithm and configuration
embedding representation
internal activations / gradients

This gives a useful spectrum:

decision-based
      ↓
score-based
      ↓
algorithm-aware
      ↓
white-box internals

OpenHalDet makes a related distinction when comparing black-box, gray-box and white-box hallucination detectors under different model-access regimes.[7]

Capability

What may the attacker change?

claim text
retrieved evidence
source ordering
evidence duplication
segmentation
rank
threshold configuration
upstream retrieval

A user may control only the prompt or claim.

A malicious document may influence retrieval.

A developer controls rank and chunking.

A vendor update may change model behavior without anybody intentionally attacking the system.

Budget

How much pressure may the attacker apply?

number of detector queries
candidate substitutions
text edits
evidence additions
retrieval attempts
compute / wall-clock budget

These distinctions tell us what kind of result we have found.

A counterexample can be:

a realistic security threat

a robustness stress test

or

a scientific falsification of an over-broad detector claim

Those are all valuable.

They are not the same claim about production risk.

2. Separate baseline errors from attack-induced failures

A subtle mistake can make an adversary look stronger than it is.

Suppose an unsupported example already passes the detector:

threshold          = 0.20
original score     = 0.12
original decision  = ACCEPT

An attack changes the score to:

adversarial score  = 0.11

The adversarial example is still a false acceptance.

But the attacker did not cause the decision failure.

The detector was already wrong.

So adversarial reporting should distinguish at least three quantities:

baseline failure
attack-induced failure
worst-case adversarial failure

Let the decision rule be:

False-acceptance attack

The adversary searches for an unsupported adversarial example:

False-rejection attack

The symmetric attack preserves support while making the example look risky:

supported claims
    ↓
artificially high risk
    ↓
review overload
    ↓
coverage collapse
    ↓
alert fatigue
    ↓
human rubber-stamping or system abandonment

Therefore report both directions.

And freeze $\tau$ before the adversarial test.

If the threshold changes after seeing attacks, the experiment has become detector tuning.

The overall workflow for a clean adversarial test is:

    graph TD
    A[one successful flip] --> B[attack-induced success]
    B --> C[report ASR]
    C --> D[freeze threshold before test]
  

Attack success is meaningful only when the threshold is fixed before measurement; otherwise the adversarial test becomes another tuning loop.

3. Organize attacks around what the measurement consumes

A long list of attack names is difficult to reason about.

A more useful structure mirrors the measurement itself.

For Hallucination Energy:

c   = claim
E   = evidence set
φ   = embedding representation
g   = evidence construction / segmentation
r   = retained rank

Policy later adds calibration and thresholding.

That gives us three broad attack families.

The difficulty ladder from Chapter 6 now becomes search pressure inside those families:

LEVEL 0  unrelated mismatch
LEVEL 1  dataset-native mismatch
LEVEL 2  same-domain mismatch
LEVEL 3  semantic-neighbor mismatch
LEVEL 4  detector-aware search
LEVEL 5  structural adversary

The important progression is not the label number.

It is that each step removes another shortcut.

Adversarial difficulty should progressively preserve what the detector sees while changing what the detector is supposed to decide.

4. Content attacks: preserve the semantic material, change the proposition

The cleanest attack changes the truth-bearing structure while preserving almost everything a broad semantic representation is likely to preserve.

Relation inversion

Evidence:
Company A acquired Company B.

Supported:
Company A acquired Company B.

Adversarial:
Company B acquired Company A.

The entities are identical.

The event vocabulary is identical.

The role assignment changed.

Polarity reversal

Evidence:
The study did not find a significant reduction in mortality.

Adversarial:
The study found a significant reduction in mortality.

One small edit can reverse the proposition while leaving lexical overlap almost perfect.

Quantity mutation

Evidence:
Revenue increased by 12%.

Adversarial:
Revenue increased by 21%.

Temporal inversion

Evidence:
Regulatory approval occurred after the trial ended.

Adversarial:
Regulatory approval occurred before the trial ended.

Causal reversal

Evidence:
A reduction in demand caused prices to fall.

Adversarial:
Falling prices caused demand to decline.

These transformations share one principle:

Change the truth-bearing structure while preserving the representation-bearing surface.

Binding attacks

Relation inversion is one special case of a deeper problem.

Suppose the evidence says:

Alice founded Alpha.
Bob founded Beta.

Now assert:

Alice founded Beta.

Every semantic ingredient is present:

Alice
Bob
Alpha
Beta
founded

The problem is which pieces are bound together.

The evidence encodes:

entity from passage 1
+
relation from passage 2
+
quantity from passage 3
+
wrong binding

This is particularly dangerous in multi-document RAG.

More retrieval creates more legitimate semantic components.

That can enlarge the space of unsupported propositions that can be assembled from individually real pieces.

More relevant evidence can increase the space of plausible recombinations.

Measure attack minimality

A structural counterexample becomes more diagnostic when the edit is small.

Record:

tokens changed
characters changed
entities preserved
relation vocabulary preserved
surface syntax preserved
embedding similarity before / after

Embedding similarity here is a diagnostic, not the oracle.

If a one-token negation flips truth while the detector score barely changes, that is stronger evidence of a missing representational distinction than a complete rewrite.

5. Context attacks: change the evidence while keeping the claim fixed

A claim–evidence detector assumes that the evidence package is an appropriate reference.

Attack that assumption directly.

Nearest-wrong evidence

Two Certum procedures build a wrong evidence set for a claim.

hard_mined_v2 is the semantic-neighbour version. For each claim it approximately performs:

    graph TD
    C[claim] --> R[rank evidence sets by centroid similarity]
    R --> SH[shortlist top K = 16]
    SH --> EX[remove the correct evidence and any set sharing its source page]
    EX --> SEL[select the most similar remaining mismatch]
  

This is the version Chapters 5 and 6 report. It picks a plausible wrong reference, but it does not consult the detector.

hardest_energy_mined is the detector-aware version. It computes Hallucination Energy for each shortlisted mismatch and selects the minimum:

Evidence inflation: permissiveness attack

Add evidence directions that help explain an unsupported claim.

For the full span:

An adversary can search for $D$ such that:

Evidence rotation: false-rejection attack

Truncated SVD is different.

When only the top $r$ directions are retained, added passages can change singular values and rotate which directions survive.

Now a supported claim can become less contained because the fixed-capacity basis has been redirected toward distractors.

That is a separate attack:

inflation attack
→ helps bad claim
→ exists even for full span if added evidence aligns with bad claim

rotation attack
→ hurts good claim
→ depends on truncated representation / capacity allocation

Support deletion

Remove the one passage that carries the decisive relationship while leaving surrounding topical material intact.

before:
20 passages, one states A acquired B

after:
19 passages still discuss A, B and the transaction
but the decisive support is gone

The detector should become more skeptical.

If the score barely changes, the measurement may be tracking topic containment rather than support-bearing content.

Evidence echo

Append a passage that simply repeats the candidate claim.

Containment should improve.

That is not a containment bug.

The reference now literally contains the assertion.

The failure is elsewhere:

Was this passage independent evidence?
Where did it come from?
Did the generated claim create the source that is now being used to justify it?

So evidence echo demonstrates:

Containment is not provenance.

Evidence duplication

Repeat one evidence passage several times.

For a full span, duplication adds no new direction.

For truncated SVD, repeated rows change singular values and can alter which directions dominate the retained basis.

A large score movement under duplication reveals sensitivity to evidence weighting rather than new evidential content.

Distractor injection

Add topically relevant but propositionally irrelevant passages.

Long-context RAG systems routinely produce this situation without an attacker.

A reliability system should not become more confident merely because the evidence package became longer and semantically broader.

6. Configuration and pipeline attacks: break the measurement around the metric

Some of the most realistic failures do not mutate the claim at all.

They change the conditions under which the measurement is computed.

Segmentation

Represent the same source as:

whole document
sentences
50-token chunks
200-token chunks
atomic propositions

The evidence matrix changes.

Its spectrum changes.

Its effective rank changes.

The score can change even though the source facts do not.

Rank

For a fixed nested SVD basis, Chapter 5 established:

r = 1, 2, 4, 8, 16, 32, full

and ask where supported and unsupported distributions converge.

Rank is not merely an implementation knob.

It controls how permissive the proxy can become.

Embedder transfer

Run the same attack corpus through several embedding models.

A failure that disappears may be representation-specific.

A failure that persists across sentence encoders gives stronger evidence of a broader limitation.

Threshold reuse

A calibrated threshold is also an attack surface when it is reused outside the distribution that justified it.

Conceptually:

calibrate on Wikipedia-like factual containment
        ↓
reuse unchanged in legal relational domain
        ↓
distribution / mechanism shift
        ↓
policy behavior no longer matches claimed risk budget

This is often an accidental deployment attack rather than a malicious one.

The test is simple: freeze the threshold from domain A and report FAR/SAR on domain B rather than silently recalibrating.

Attack the verifier chain

A complete verifier may contain:

D = decomposition
R = retrieval
A = attribution
S = support judgment
P = provenance
G = aggregation

Each stage has a natural attack:

PROBE makes a closely related process-level move by evaluating claim decomposition, evidence finding, evidence evaluation and hallucination localization separately rather than reducing the verifier to one judge call.[6]

A useful red-team record therefore asks two questions:

Which stage failed first?

Which later failures were merely consequences of that corrupted state?

For example:

retrieval returns wrong evidence
        ↓
attribution faithfully links to wrong evidence
        ↓
support classifier correctly judges that evidence
        ↓
aggregator produces wrong final decision

Only the first stage may have been directly compromised.

The rest is error propagation.

That distinction prevents us from blaming the containment sensor for a retrieval failure—or praising downstream stages that never received the right evidence.

7. Search pressure should produce a curve, not one hard-negative score

Once an attack searches over alternatives, the search budget becomes part of the result.

The historical result used a fixed shortlist of 16 and similarity-based selection. The Adversarial Degradation Curve from Chapter 6 §14 is the artifact that generalizes it: freeze the detector, embedder, rank, segmentation, threshold, candidate pool, and claim set, then sweep the budget.

For Hallucination Energy the sweep is concrete. For each:

K = 1, 4, 16, 64, 256

construct the shortlist and select the energy-aware mismatch:

    graph LR
    SW[Freeze detector, embedder, rank, segmentation, threshold, candidate pool, claim set]
    SW --> SWEEP[K = 1, 4, 16, 64, 256]
    SWEEP --> REPORT[Report ROC-AUC, gap, ASR, FAR, SAR vs log2 K]
    REPORT --> CURVE[Adversarial Degradation Curve]
  

The sweep treats attack budget as the independent variable, so the important result is how detector behavior degrades as harder negatives become available.

8. Metamorphic testing gives us expected relationships

Many hallucination tests have an oracle problem because open-ended language has no single exact output.

Metamorphic testing helps by specifying a metamorphic relation: how a measurement should change, or should not change, after a controlled transformation.[1][2] The two families below are the standard invariance and directional relations from that literature.

Let $T$ be a transformation.

Define the score effect:

The support label should remain unchanged.

We expect:

Examples:

meaning-preserving paraphrase
consistent fictional entity renaming in claim and evidence
evidence reordering
formatting changes

Sensitivity transformations

The truth/support relation changes in a known direction.

For a higher-is-riskier detector we expect:

Examples:

subject/object swap
negation
quantity mutation
temporal reversal
remove sole support
replace correct evidence with nearest wrong evidence

This gives a compact test matrix:

    graph TD
    subgraph Invariance
        PAR[Paraphrase] --> STAB[score approximately stable]
        REN[Consistent entity renaming] --> STAB
        ORD[Evidence reordering] --> STAB
    end
    subgraph Sensitivity
        SWAP[Subject/object swap] --> INC[risk should increase]
        NEG[Negation] --> INC
        QUANT[Quantity mutation] --> INC
        TEMP[Temporal reversal] --> INC
        REM[Remove sole support] --> INC
    end
  

Metamorphic testing defines reliability through expected relationships between transformed cases rather than through a single exact reference answer.

One transformation deserves special treatment:

append the candidate claim verbatim to the evidence

Containment should improve because the evidence representation now contains the claim.

If provenance says the new passage is circular or untrusted, policy may still reject it.

This is therefore a scope probe, not evidence that the containment measurement failed.

The same metamorphic idea will return in Chapter 9 when we treat sensitivity as a broader reliability signal rather than only an attack technique.

9. Oracle discipline: attack success is conditional on attack validity

A red team can accidentally optimize ambiguity instead of vulnerability.

Structural examples are especially vulnerable to this.

Suppose the source says:

A may increase B under some conditions.

and the attack says:

A does not increase B.

Is that a clean contradiction?

Not necessarily.

The modality makes the label ambiguous.

So structural attacks should prefer oracle-controlled construction.

A structured fact can be represented as:

fact = {
    "subject": "Company A",
    "relation": "acquired",
    "object": "Company B",
    "date": "2024-03-14",
    "polarity": "positive",
}

Then render the supported statement and apply controlled transformations.

For every generated attack, record whether it is actually valid.

Define:

10,000 attempted transformations
9,420 oracle-valid examples
6,110 successful detector evasions

from a method that produces only 4,000 valid examples out of 10,000 attempts.

The first quantity measures attack generation yield.

The second measures detector vulnerability.

Do not collapse them.

And do not use semantic similarity as the ground-truth oracle for structural attacks.

High semantic similarity is often exactly the property we are trying to preserve while truth changes.

The adversary should attack the detector, not the annotation procedure.

10. Freeze both sides—and keep three adversarial suites separate

Adaptive red teaming creates a familiar evaluation trap.

Suppose we:

attack detector v1
observe failure
change detector
add attack to benchmark
attack again
change detector again

Eventually the detector may achieve 100% on the accumulated failures.

That tells us it remembers known failures.

It does not prove it generalizes adversarially.

So separate three artifacts.

    graph TD
    AD[ATTACK DEVELOPMENT: invent / tune transformations, inspect failures] --> FREEZE[FREEZE ATTACKER]
    FREEZE --> HOT[HELD-OUT ADVERSARIAL TEST: new examples / entities / sources]
    HOT --> MIT[MITIGATE]
    MIT --> REG[KNOWN FAILURE ENTERS REGRESSION]
    REG --> NOVEL[NOVEL-MECHANISM SEARCH CONTINUES]
  

A failure can graduate into regression coverage only after the attacker is frozen and a held-out adversarial test has separated remembered cases from new pressure.

Known-failure regression suite

Question:
Did the detector forget a failure we already discovered?

Every confirmed counterexample belongs here.

Held-out adversarial generalization suite

Question:
Does the mitigation generalize to unseen entities,
sources and examples of the SAME failure mechanism?

The attacker is frozen before this test.

Novel-mechanism red team

Question:
Can we discover a DIFFERENT blind spot?

This is where adaptive exploration belongs.

Record:

detector version
attacker version
attack-development corpus
held-out adversarial corpus
query budget
random seed
oracle method
threshold version

The principle is worth preserving:

A test stops being a test when its result changes what you try next.

11. Preserve counterexamples as executable evidence about the detector

A counterexample is wasted if it appears once in a notebook and disappears.

Store it with the relationship the future detector is expected to preserve.

counterexample = {
    "attack_id": "binding_swap_0042",
    "detector": "hallucination_energy",
    "detector_version": None,
    "failure_family": "content",
    "failure_mechanism": "role_binding",
    "attack_objective": "false_acceptance",
    "attacker_access": "score_based",
    "oracle_label": "UNSUPPORTED",
    "oracle_method": "structured_relation_swap",
    "oracle_version": None,
    "source_evidence": [],
    "original_claim": "Company A acquired Company B.",
    "adversarial_claim": "Company B acquired Company A.",
    "pre_attack_score": None,
    "post_attack_score": None,
    "delta_score": None,
    "threshold": None,
    "attack_budget": 1,
    "edit_distance_tokens": 2,
    "validity": None,
    "realism": "synthetic_capability_probe",
    "expected_relation": "post_attack_risk_should_increase",
    "expected_policy": "REVIEW_OR_REJECT",
    "first_compromised_stage": "measurement",
    "first_failed_version": None,
    "status": "ACTIVE_REGRESSION",
}

Possible statuses include:

ACTIVE_REGRESSION
MITIGATED
ACKNOWLEDGED_LIMITATION

Not every proxy boundary should be hidden by increasingly complex patches.

Sometimes the correct response is to record:

this sensor does not measure that property.

The regression artifact should therefore test relationships, not only exact score values.

A new embedder or recalibration may legitimately move absolute scores.

The durable expectation may simply be:

negating the decisive proposition
must increase risk

or:

consistent fictional renaming
should not materially change support score

This makes the corpus more robust to future implementation changes.

A red-team report should aggregate these records:

red_team_report = {
    "victim_version": None,
    "threshold_version": None,
    "attacker_version": None,
    "attack_families": ["content", "context", "configuration_pipeline"],
    "budgets": [1, 4, 16, 64, 256],
    "attempted_attacks": None,
    "valid_attacks": None,
    "validity_rate": None,
    "baseline_far": None,
    "baseline_frr": None,
    "induced_asr_false_accept": None,
    "induced_asr_false_reject": None,
    "held_out_results": None,
    "first_compromised_stage_counts": None,
    "regression_ids": [],
}

That turns adversarial testing into an auditable engineering artifact rather than a collection of anecdotes.

12. The current Hallucination Energy results already show three different failure regimes

Our existing experiments — all using the semantic-neighbour hard_mined_v2 negatives, not the energy-aware adversary — contain three qualitatively different outcomes.

Wikipedia / FEVEROUS-style

Hard-mined mismatches retained substantial mean separation:

supported mean      ≈ 0.3714
hard-negative mean  ≈ 0.6950
gap                 ≈ +0.3236

Containment still carried useful signal.

But strict low-FAR operation accepted few supported examples.

This is primarily an operating-point problem until stronger attacks prove otherwise.

PubMed

The hard-negative gap remained similarly positive:

supported mean      ≈ 0.3700
hard-negative mean  ≈ 0.6944
gap                 ≈ +0.3244

Again, semantic-neighbour mismatches did not erase the signal.

CaseHOLD

The ordering reversed:

supported mean      ≈ 0.6680
hard-negative mean  ≈ 0.6075
gap                 ≈ -0.0605

This is a different kind of result. On this relation-heavy legal task the detector inverts: it scores the structurally wrong holdings as more contained, on average, than the correct ones. A threshold cannot fix a sign flip. Chapter 8 §15 explains why.

The hard negatives can look more contained than the supported examples.

Now the question is not:

Which threshold should we choose?

It is:

Which distinction is missing from the representation or proxy?

That is exactly the result adversarial evaluation should search for:

a regime where proxy and target stop moving in the expected direction.

The historical hard_mined_v2 result is a Level-3 semantic-neighbour attack at $K\le16$. It does not yet exercise the energy-aware selection the Certum code already supports, and it does not yet prove robustness across search pressure.

The next concrete experiments are therefore obvious:

energy-aware mining (hardest_energy_mined) at matched conditions
K-budget degradation sweep
structured binding / relation corpus
support-deletion metamorphic tests
false-rejection paraphrase + fragmentation tests
rank / segmentation attack matrix
frozen-threshold domain transfer

Those are research tasks, not numbers we should invent in prose.

13. A compact red-team protocol

The chapter can now be reduced to a repeatable procedure.

    graph TD
    F[1 — Freeze and classify: victim version, threshold, security vs stress vs falsification]
    O[2 — Declare the oracle and attack surface: what makes it supported/unsupported, attacker knowledge, capabilities, budget]
    B[3 — Establish baseline errors: un-attacked FAR/FRR]
    P[4 — Apply pressure across three families: content, context, configuration/pipeline]
    V[5 — Validate every attack: attempted count, validity rate, ASR conditional on validity]
    H[6 — Freeze the attacker and test held out]
    A[7 — Attribute, preserve, and continue: first compromised stage, regression artifact, acknowledged boundary]
  

The protocol makes adversarial evaluation reproducible by freezing the victim and attacker, measuring valid attacks, and preserving failures without confusing regression with discovery.

1 — Freeze and classify

Declare:

victim version
threshold / calibration
security threat vs stress test vs scientific falsification

2 — Declare the oracle and attack surface

Specify:

what makes the candidate supported / unsupported
what the attacker may know
what the attacker may change
what budget applies

3 — Establish baseline errors

Measure the un-attacked FAR/FRR before giving the attacker credit.

4 — Apply pressure across the three families

CONTENT
role / polarity / quantity / time / binding

CONTEXT
wrong neighbor / deletion / inflation / echo / duplication / distractors

CONFIGURATION + PIPELINE
segmentation / rank / embedder / threshold / retrieval / attribution / aggregation

Increase search budget where appropriate.

5 — Validate every attack

Report:

attempted count
oracle-valid count
validity rate
attack success conditional on validity
score effect distribution

6 — Freeze the attacker and test held out

Separate attack development from final adversarial evidence.

7 — Attribute, preserve, and continue

Record:

first compromised stage
propagated consequence
regression artifact
held-out generalization result
acknowledged proxy boundary if not fixable

Then start searching for the next mechanism.

The objective is not to finish red teaming.

It is to make the system increasingly explicit about what it can and cannot guarantee.

14. What breaking the detector actually teaches us

The purpose of adversarial evaluation is not to prove that every detector is bad.

Any finite sensor can be pushed outside its contract.

The useful result is more precise.

Breaking a detector tells us which statement about it was too strong.

For Hallucination Energy:

WRONG:
low energy means true

STILL WRONG:
low energy means supported

CLOSER, BUT TOO STRONG:
low energy means the evidence entails the proposition

MEASUREMENT-SAFE CLAIM:
low energy means the claim embedding is well represented
by the configured evidence-derived subspace

The final statement survives because it is the actual measurement definition.

The attacks then tell us what must exist around it:

containment
    +
relation / binding checks
    +
polarity / quantity / temporal structure
    +
provenance
    +
verification
    +
policy

Sometimes a red team finds an implementation bug.

Fix it.

Sometimes it finds a calibration or pipeline bug.

Repair the system.

Sometimes it finds a boundary in the observable itself.

Do not pretend calibration can recover information the sensor never represented.

When the implementation is correct, adversarial evaluation narrows our interpretation of the measurement until the interpretation matches what the measurement can actually support.

A successful red team turns an overclaim into a boundary.

That is not a defeat for the metric.

It is progress in understanding it.

Research roots

This chapter combines our Certum hard-negative work with broader research on dynamic benchmarks, difficult hallucinations, process-level verification and metamorphic testing.

1. Steven Cho, Stefano Ruberto and Valerio Terragni, “Metamorphic Testing of Large Language Models for Natural Language Processing,” 2025. Surveys 191 metamorphic relations and implements 36 representative relations across roughly 560,000 LLM tests, demonstrating how transformation relations can expose failures when exact output oracles are difficult. https://arxiv.org/abs/2511.02108

2. Borui Yang, Md Afif Al Mamun, Jie M. Zhang and Gias Uddin, “Hallucination Detection in Large Language Models with Metamorphic Relations,” 2025. Introduces MetaQA, using prompt mutations and metamorphic relations for hallucination detection without requiring external factual resources. https://arxiv.org/abs/2502.15844

3. Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda and Pascale Fung, “HalluLens: LLM Hallucination Benchmark,” ACL 2025. Includes dynamic test-set generation intended to mitigate leakage and strengthen robustness. https://aclanthology.org/2025.acl-long.1176/

4. Shrey Pandit, Jiawei Xu, Junyuan Hong, Zhangyang Wang, Tianlong Chen, Kaidi Xu and Ying Ding, “MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models,” EMNLP 2025. Builds 10,000 PubMedQA-derived examples through a controlled hallucination pipeline and reports that harder hallucinations are semantically closer to ground truth and harder for detectors to identify. https://aclanthology.org/2025.emnlp-main.143/

5. Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi, Elaine Wong and Leman Akoglu, “Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights,” ACL 2026. Highlights gaps in realistic long-context RAG evaluation and label-noise stress testing. https://aclanthology.org/2026.acl-long.680/

6. Yu Zhang, Peter Belcak, Shizhe Diao, Yonggan Fu, Shaona Ghosh, Morteza Mardani, Eileen Margaret Peters Long, Bei Yu and Pavlo Molchanov, “PROBE: PROcess-Based BEnchmark for Hallucination Detection,” Findings of ACL 2026. Decomposes hallucination detection into claim decomposition, evidence finding, evidence evaluation and hallucination localization, enabling stage-level diagnosis. https://aclanthology.org/2026.findings-acl.2099/

7. Xinyi Li et al., “OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios,” 2026. Standardizes evaluation across diverse generation settings and detector families under black-box, gray-box and white-box access regimes. https://arxiv.org/abs/2606.06959

8. Ernan Hughes, Certum, open-source implementation of claim–evidence geometry, hard-negative mining, calibration, policy gating and evaluation. The hard_mined_v2 procedure shortlists up to 16 centroid-similar evidence sets and selects the most similar mismatch (with a source-page and element-id leakage guard); a separate hardest_energy_mined mode instead selects the shortlisted mismatch with minimum Hallucination Energy. https://github.com/ernanhughes/certum

Next: Containment Is Not Truth

We started Chapter 5 with a geometric hypothesis.

We evaluated it in Chapter 6.

Chapter 7 has now deliberately attacked it under controlled pressure.

The attacks expose a recurring pattern:

all semantic ingredients can be present

while

the proposition is still wrong

A false claim can preserve:

entities
topic
vocabulary
semantic neighborhood
evidence-span membership

while changing:

role
polarity
quantity
time
causal direction
binding

That is no longer merely an evaluation issue.

It is a statement about what containment means.

Chapter 8 takes that adversarial result and asks the deeper first-principles question:

Why can a claim be completely contained by its evidence representation and still be false?

The answer will force us to separate semantic material from propositional structure—and to stop expecting one scalar to carry information that the representation never encoded.