Hallucination From First Principles: OK on Programmer.ie: Modern AI programming

When Models Make Things Up

Sat, 29 Aug 2026 22:10:00 +0100

A language model can produce an answer that is fluent, grammatical, detailed, confident, and wrong.

That combination is what makes hallucination interesting.

Ordinary software failures usually leave evidence. A parser throws an exception. A database rejects an invalid constraint. A test fails. A network call times out. The failure changes the shape of the result.

A hallucinating language model can do something more difficult to handle: it can fail without looking broken.

Hallucination Is Not One Thing

Sat, 29 Aug 2026 22:45:00 +0100

Chapter 1 began with a familiar failure: a language model says something fluent that it cannot justify.

That description is useful until we try to build a detector.

Consider six outputs:

A. The model invents a person who does not exist.

B. The model names two real companies but reverses which acquired which.

C. The model cites a real paper that does not support the claim.

D. The model says it inspected a file that it never received.

E. The model answers a question even though the available evidence is insufficient.

F. The model gives the same generic recommendation after the decisive facts
   of the problem have been reversed.

All six can produce plausible text.

Evidence, Truth, and Verifiability

Sat, 29 Aug 2026 23:20:00 +0100

Chapter 2 gave us a rule for talking about hallucination precisely:

Relative to what reference did the output fail?

That question immediately creates another one.

What counts as a reference strong enough to constrain the answer?

Suppose a model says:

Company A acquired Company B in 2024.

There are several different things we might mean when we ask whether this is acceptable:

What exact proposition has been asserted?

Is that proposition true at the relevant time?

Do we possess evidence for it?

Does that evidence actually support this exact relationship?

Can we identify the source and passage that support it?

Where did that source come from?

Is the source sufficiently reliable and current for this claim?

Is the evidence package sufficient for the action we want to authorize?

Could the claim be verified by this system now?

Has the verification actually been performed?

Would the active policy allow the claim to be published or acted upon?

Those questions are related.

How Do You Measure a Hallucination?

Sat, 29 Aug 2026 23:14:00 +0100

The first three chapters deliberately avoided building a detector.

Before designing one, we needed to define exactly what it would be expected to detect.

Chapter 1 established that fluent generation can continue after evidential support has weakened or disappeared.

Chapter 2 showed that hallucination covers several different failure relationships.

Chapter 3 then separated the objects a reliability system must not collapse:

truth
≠
evidence
≠
support
≠
attribution
≠
provenance
≠
verification
≠
policy acceptance

Now we can finally ask the engineering question:

Hallucination Energy

Sat, 29 Aug 2026 23:14:00 +0100

Chapter 4 ended with a rule:

A hallucination measurement is a sensor, not a verdict.

This chapter builds one such sensor completely.

It begins with a property we care about:

semantic containment

But the first lesson of Chapter 4 was the proxy gap. We cannot inspect semantic containment directly. We can inspect representations.

So the construction in this chapter has three layers:

    graph TD
    LP[LATENT PROPERTY: semantic containment] --> RA[representation assumption]
    RA --> PP[PROXY PROPERTY: embedding-subspace containment]
    PP --> GC[exact geometric calculation]
    GC --> M[MEASUREMENT: squared projection residual]

That distinction is the chapter’s most important caveat.

How to Evaluate a Hallucination Detector

Sat, 29 Aug 2026 23:14:00 +0100

Chapter 5 gave us something concrete enough to evaluate.

Hallucination Energy has:

a target property
an observable proxy
a mathematical definition
an implementation
continuous outputs
natural-task signal
adversarial failures

That sounds like progress.

It is also the point where it becomes dangerously easy to fool ourselves.

Suppose a detector reports:

AUC = 0.75

Is that good?

Perhaps.

But we still do not know:

which class was called positive
how hard the negatives were
whether train and test shared source documents
whether they shared entities or semantic neighborhoods
what prevalence the deployment actually has
whether the labels were trustworthy
how wide the confidence interval was
what happens at 1% false acceptance
how uncertain that 1% estimate is
whether the threshold transfers to another domain
whether the score is a probability at all
whether the verifier itself failed upstream
whether the generator changed
whether decoding parameters changed
whether the model collapses on structural adversaries

A single number cannot answer those questions.

Breaking the Detector

Sun, 30 Aug 2026 00:17:00 +0100

Chapter 6 changed the standard for evaluation.

A detector is not trustworthy because it has a good AUC.

It becomes trustworthy only after we understand how its errors behave under the conditions in which policy will use it.

That immediately suggests a more aggressive question.

Instead of waiting for those conditions to appear naturally, can we search for them deliberately?

Suppose a containment detector works well on ordinary unsupported claims.

Containment Is Not Truth

Sun, 30 Aug 2026 09:10:00 +0100

Chapter 7 deliberately broke the detector.

The most useful failures had a peculiar shape. The unsupported claim did not introduce a completely new topic. It did not necessarily introduce a new entity. It did not always wander outside the semantic region represented by the evidence.

Instead, the attack often preserved almost everything that a broad semantic representation could reasonably notice:

same entities
same event vocabulary
same topic
same evidence neighborhood
same quantities or dates, sometimes
same semantic ingredients

while changing the thing that made the proposition correct:

Beyond Hallucination: Consistency and Sensitivity

Sun, 30 Aug 2026 10:16:00 +0100

Chapter 8 ended with a rule:

If a downstream decision depends on a distinction, do not discard that distinction before the decision is made.

That rule breaks the idea of one universal hallucination score.

A response can be well contained and still reverse a relation. It can preserve every relation and still ignore the decisive facts of the problem. It can be correct once and unstable under a harmless rephrasing. It can be perfectly repeatable and consistently wrong.

The Safe but Useless Model

Sun, 30 Aug 2026 10:42:00 +0100

Chapter 9 changed the unit of evaluation.

Instead of asking whether one answer looks good, we began studying a family of related executions.

That immediately reveals a failure that static evaluation can miss almost completely.

Consider two organizations.

ORGANIZATION A
3 months of runway
falling demand
negative cash flow
credit line nearly exhausted

ORGANIZATION B
5 years of runway
rapidly growing demand
strong margins
large cash reserve

Ask both:

Knowing When Not to Answer

Sun, 30 Aug 2026 11:15:00 +0100

Chapter 10 ended with an ambiguity.

A model gave weak, generic advice.

Perhaps it ignored decisive evidence.

But perhaps the evidence never justified a decisive answer in the first place.

Those are not the same failure.

Consider this question:

What was Company A's Q3 revenue?

and this evidence:

Q1 revenue was $41.2 million.
Q2 revenue was $43.8 million.
The company expects second-half demand to improve.

A language model can easily produce something that sounds reasonable:

From Measurements to Policy

Sun, 30 Aug 2026 18:37:00 +0100

Chapter 11 ended with a typed reliability record.

It might say:

containment          = PASS
relation_fidelity    = PASS
sensitivity          = PASS
epistemic_adequacy   = ANSWERABLE
provenance           = UNVERIFIED

That record describes the candidate.

It still does not authorize the candidate.

To make the distinction concrete, we built a deliberately flawed reference policy that checked containment, structural fidelity, and answerability but forgot to require verified provenance for a high-risk action.

Running the same immutable record through two policy versions produced:

Verification, Repair, and Rejection

Sun, 30 Aug 2026 18:38:00 +0100

Chapter 12 gave the system a control plane.

It can now say:

commitment = HOLD
next_action = VERIFY

or:

commitment = HOLD
next_action = RETRIEVE

or:

commitment = HOLD
next_action = REFINE

or:

commitment = DENY
response_mode = REJECTION

Those are policy decisions.

They are not yet recovery implementations.

The obvious implementation is dangerously tempting:

The answer failed a check.
Ask the model to fix it.

But the model that produced the first unsupported statement can produce a second unsupported statement while “correcting” it.

The Memory Contamination Problem

Sun, 30 Aug 2026 18:39:00 +0100

Chapter 13 ended with a strict recovery invariant:

repair
→ new candidate state
→ re-measure
→ re-authorize

and with one additional rule:

candidate in HOLD
→ do not persist as trusted factual memory

That second rule changes the scale of the problem.

A hallucinated sentence displayed once is transient.

The same sentence written into persistent state can survive the conversation that created it.

It can then be:

retrieved
summarized
copied
cited
used as agent experience
used to fill a user profile
inserted into a vector index
fed into another model
used as a future repair source

and eventually appear to the system as if it came from somewhere else.

Building Systems That Distrust Their Models

Sun, 30 Aug 2026 21:01:00 +0100

The first chapter began with a simple observation:

A language model can produce a fluent answer
without possessing a mechanism that proves the answer is true.

Fourteen chapters later, that fact has not changed.

The model can still:

invent
misbind
misattribute
ignore decisive context
answer without enough evidence
accept bad retrieval
repair one error by creating another
repeat its own stored mistake

The final architecture does not make those possibilities disappear.

Appendix: The Evidence Ledger

Mon, 31 Aug 2026 09:00:00 +0100

The book argues that a claim should be governed by its provenance, its evidential support, and an explicit statement of what it does not establish. This appendix applies that discipline to the book itself.

Every original empirical number produced for this book is listed here with three things: where it was measured, what its provenance status is, and what it does not establish.

Two facts are visible immediately from the ledger: