Metamorphic Testing on Programmer.ie: Modern AI programming

Breaking the Detector

Sun, 30 Aug 2026 00:17:00 +0100

Chapter 6 changed the standard for evaluation.

A detector is not trustworthy because it has a good AUC.

It becomes trustworthy only after we understand how its errors behave under the conditions in which policy will use it.

That immediately suggests a more aggressive question.

Instead of waiting for those conditions to appear naturally, can we search for them deliberately?

Suppose a containment detector works well on ordinary unsupported claims.

Beyond Hallucination: Consistency and Sensitivity

Sun, 30 Aug 2026 10:16:00 +0100

Chapter 8 ended with a rule:

If a downstream decision depends on a distinction, do not discard that distinction before the decision is made.

That rule breaks the idea of one universal hallucination score.

A response can be well contained and still reverse a relation. It can preserve every relation and still ignore the decisive facts of the problem. It can be correct once and unstable under a harmless rephrasing. It can be perfectly repeatable and consistently wrong.

The Safe but Useless Model

Sun, 30 Aug 2026 10:42:00 +0100

Chapter 9 changed the unit of evaluation.

Instead of asking whether one answer looks good, we began studying a family of related executions.

That immediately reveals a failure that static evaluation can miss almost completely.

Consider two organizations.

ORGANIZATION A
3 months of runway
falling demand
negative cash flow
credit line nearly exhausted

ORGANIZATION B
5 years of runway
rapidly growing demand
strong margins
large cash reserve

Ask both: