From Measurements to Policy

Chapter 11 ended with a typed reliability record.

It might say:

containment          = PASS
relation_fidelity    = PASS
sensitivity          = PASS
epistemic_adequacy   = ANSWERABLE
provenance           = UNVERIFIED

That record describes the candidate.

It still does not authorize the candidate.

To make the distinction concrete, we built a deliberately flawed reference policy that checked containment, structural fidelity, and answerability but forgot to require verified provenance for a high-risk action.

Running the same immutable record through two policy versions produced:

policy-v1  → commitment=PERMIT  next_action=NONE
policy-v2  → commitment=HOLD    next_action=VERIFY

Nothing about the candidate changed.

Nothing about the evidence changed.

Nothing about the measurements changed.

The policy changed.

That is the subject of this chapter.

Measurements describe the state of the candidate. Policy decides what that state authorizes.

Environment

The small reference policy engine in this chapter was executed with:

Python 3.13.5
standard library only
no external services

The examples are deterministic policy tests over synthetic reliability records. They are not model-quality experiments.

Where we are

The book has gradually separated concepts that are often collapsed together:

truth
≠ evidence
≠ support
≠ provenance
≠ containment
≠ structural fidelity
≠ consistency
≠ sensitivity
≠ epistemic adequacy
≠ authorization

The first eleven chapters mostly asked:

What happened?
What did we observe?
How trustworthy is that observation?
What information is missing?

Chapter 12 asks a different question:

Given those observations, what is the system allowed to do?

This is where reliability becomes control flow.

1. Measurement is not diagnosis, authorization, routing, or execution

Chapter 4 established that the sensor is not the verdict.

We can now make the full chain explicit:

    graph TD
    RO[RAW OBSERVATION: energy = 0.184] --> DI[DIAGNOSTIC INTERPRETATION: containment_state = WITHIN_CALIBRATED_REGION, calibration = he-cal-v7]
    DI --> RR[RELIABILITY RECORD: provenance = UNVERIFIED, epistemic_adequacy = ANSWERABLE]
    RR --> PD[POLICY DECISION: commitment = HOLD, next_action = VERIFY]
    PD --> EN[ENFORCEMENT: source verification runs]
    EN --> ER[EXECUTION RECORD: what actually happened]
  

These are different objects.

A diagnostic PASS means:

the sensor satisfied its own declared diagnostic contract.

It does not mean:

the application authorizes the candidate.

Likewise, low Hallucination Energy does not mean:

publish the claim
send the payment
commit the memory
approve the deployment

Strong entailment does not establish source authority.

Verified provenance does not establish freshness.

Even a completely clean reliability record may encounter a policy that prohibits the requested action.

Never let a sensor silently become a permission system.

The canonical architecture is therefore:

measurement
→ diagnostic interpretation
→ reliability record
→ policy
→ enforcement

2. There are several policy scopes

The same architectural principle appears at different commitment boundaries.

Candidate policy

May this claim be asserted?
May this paragraph be shown?
May this memory item be committed?

Execution policy

May this tool call execute?
May this payment be sent?
May this database mutation occur?

Release policy

May this model version reach production?
May this workflow be promoted?

These scopes should not automatically share schemas, thresholds, or action vocabularies.

But they share the same pattern:

evidence
→ structured record
→ policy
→ trusted enforcement

This also clarifies how external governance work relates to this chapter. Audit-as-Code, for example, applies versioned machine-checkable gates primarily at assurance and release scope.[3] The architecture is analogous to runtime candidate policy, but the control boundary is different.

3. Use one canonical policy interface

We do not need four overlapping mathematical definitions of policy.

Let:

Let:

domain
risk tier
action type
jurisdiction
latency budget
human-review capacity
required measurement portfolio

And let:

step index
retrieval attempts
verification attempts
clarification attempts
previous routes

Use two policy functions.

Commitment policy

May the candidate cross the current commitment boundary?

Commitment is the term for the assertion-scoped decision — may this content be stated. Authorization (Chapter 15) is the same kind of decision at the action boundary — may this side effect execute. They are not synonyms; they gate different boundaries, and a candidate can be committed as an assertion while the action it proposes is not authorized.

Routing policy

What should happen next?

This immediately resolves an ambiguity from simpler designs.

commitment = HOLD
next_action = VERIFY

means:

the candidate is not authorized yet
but source verification is authorized.

The old representation:

{
    "authorized": False,
    "action": "VERIFY",
}

was ambiguous because VERIFY itself is an authorized action.

4. A policy decision is not one flat Action enum

A single enum such as:

ACCEPT
RETRIEVE
VERIFY
REVIEW
ABSTAIN
REJECT

mixes different kinds of state.

ACCEPT describes commitment.

RETRIEVE describes recovery.

REVIEW describes escalation.

ABSTAIN describes response disposition.

REFUSE_REQUEST describes request-level prohibition.

These should be represented separately.

A useful policy decision has dimensions such as:

COMMITMENT
PERMIT | HOLD | DENY

NEXT ACTION
NONE | REFINE | RETRIEVE | VERIFY | ASK | REVIEW

RESPONSE MODE
NORMAL | QUALIFIED | PARTIAL | ABSTENTION | REFUSAL

ESCALATION
AUTO | REVIEW_REQUIRED | HUMAN_APPROVAL_REQUIRED

OBLIGATIONS
verify_provenance
state_uncertainty
omit_unverified_claim
...

For example:

policy_decision = {
    "commitment": "HOLD",
    "next_action": "VERIFY",
    "escalation": "AUTO",
    "response_mode": "NONE",
    "obligations": [
        "verify_provenance",
    ],
    "reason_codes": [
        "PROVENANCE_UNRESOLVED",
    ],
}

This also separates three superficially similar terminal states.

A hallucinated candidate can be rejected while the user request remains valid and may be regenerated.

An answer can be abstained from because evidence is inadequate.

A request can be refused even when the system knows the answer perfectly well.

Those are different failures and different control paths.

5. Hard constraints define the feasible action set

Suppose a system computes:

containment      = 0.95
consistency      = 0.93
sensitivity      = 0.90
provenance       = 0.00

A weighted average could produce:

reliability = 0.695

That number can hide the only fact that matters.

If verified provenance is mandatory for this action, the candidate cannot be committed.

So policy should first derive a feasible action set.

Let:

Then, and only then, softer costs can choose among the remaining actions:

HARD CONSTRAINTS
What is impermissible?
        ↓
FEASIBLE ACTIONS
What remains possible?
        ↓
SOFT COSTS / UTILITY
Which permissible action is best?

Never allow an average to cancel a non-negotiable failure.

This is the same broad pattern used in recent Audit-as-Code work: aggregate readiness can be useful while critical blockers independently force a blocking result.[3]

The policy objective is therefore not maximal restriction.

It is:

Choose the least restrictive action that satisfies the required constraints.

That preserves justified utility without permitting unjustified commitment.

6. Policy budgets should be explicit constraints

Policy consumes resources:

retrieval → latency + tokens
verification → compute
review → human attention
clarification → user effort
abstention → lost coverage
false acceptance → risk / harm

A useful optimization can therefore be written as:

If mandatory review demand exceeds available review capacity, the system should reduce coverage, delay, or abstain rather than silently violate the mandatory floor.

The key ordering is:

hard risk constraints first
resource optimization second

Chapter 6 placed cost trade-offs around detector operating points.

Chapter 12 places them where they ultimately belong: in policy.

7. Unknown states need their own type system

A production record may contain:

PASS
FAIL
UNCERTAIN
NOT_MEASURED
NOT_RUN
UNAVAILABLE
NOT_APPLICABLE

These states mean different things.

The classic bug is:

if state != "FAIL":
    allow()

because then:

NOT_MEASURED

becomes an accidental synonym for:

PASS.

Absence of evidence from a sensor is not positive evidence of safety.

NOT_RUN is useful in progressive systems.

If a request hits a hard prohibition before an expensive sensitivity test is executed, sensitivity was not forgotten and did not fail.

It was intentionally skipped because the route was already terminal.

That distinction matters in audits.

8. Risk tiers should preserve semantics but may require more measurements

Suppose:

provenance = UNVERIFIED

That state should mean the same thing in every application.

But different applications may require different measurement portfolios and policies.

Exploratory drafting

provenance measurement optional
unverified claim may be shown with qualification

Customer-facing factual answer

provenance required
UNVERIFIED → VERIFY

Autonomous external action

provenance required
UNVERIFIED → HOLD or DENY

So the correct rule is not:

risk tiers never change measurements.

It is:

Risk tiers should not silently change measurement semantics. They may change which measurements are required and how their states map to policy.

This preserves comparability while allowing high-risk applications to demand stronger evidence.

9. Floors apply only to ordered dimensions

Some policy outputs have a natural order.

For escalation:

AUTO
<
REVIEW_REQUIRED
<
HUMAN_APPROVAL_REQUIRED

Suppose policy defines a floor:

the executed escalation cannot be less conservative than the deterministic minimum.

Pacella, Papadia and Giliberti demonstrate this pattern in a 2026 agentic-process experiment. Among 166 onboarding profiles, 43 required mandatory human approval; one unguarded Qwen2.5 configuration failed to escalate any of those 43 cases, while the deterministic policy floor restored mandatory-approval recall to 1.0 in the evaluated dataset and configuration.[2]

But the guarantee is local.

It assumes validated inputs, correct policy, correct extraction, and trusted enforcement.

And it applies only to ordered dimensions.

ASK
RETRIEVE
VERIFY

are not naturally ordered.

They are different recovery branches.

Likewise:

floor compliance
≠
policy correctness.

A perfectly enforced bad floor is still a bad policy.

10. Precedence chooses control; obligations preserve diagnosis

Several rules may fire together.

Suppose:

policy restriction = TRUE
sensitivity        = FAIL

The request-level prohibition should determine the terminal control action.

But the trace can still record that the sensitivity rule also fired.

A useful combination strategy is:

hard DENY overrides commitment
        +
matched lower-priority diagnoses remain in trace
        +
only executable obligations compatible with the terminal route survive

For example:

{
  "commitment": "DENY",
  "next_action": "REFUSE_REQUEST",
  "matched_rule": "POLICY_PROHIBITED",
  "reason_codes": [
    "POLICY_PROHIBITED",
    "SENSITIVITY_FAIL"
  ],
  "suppressed_rules": [
    "SENSITIVITY_REVIEW"
  ],
  "obligations": []
}

The policy therefore does not destroy diagnostic information simply because one rule wins control flow.

This is safer than a long if/elif chain whose first branch erases every other fact about the candidate.

11. A small runnable policy engine

The following reference implementation deliberately keeps the policy simple enough to inspect. It is experiments/hallucination/policy_engine.py; the routes below are its actual output.

It uses:

commitment
next action
escalation
response mode
reason codes
obligations
execution history

rather than one flat action.

from dataclasses import dataclass
from enum import Enum

class Commitment(str, Enum):
    PERMIT = "PERMIT"
    HOLD = "HOLD"
    DENY = "DENY"

class NextAction(str, Enum):
    NONE = "NONE"
    REFINE = "REFINE"          # revise the candidate; executed as repair in Chapter 13
    RETRIEVE = "RETRIEVE"
    VERIFY = "VERIFY"
    ASK = "ASK"
    REVIEW = "REVIEW"
    ABSTAIN = "ABSTAIN"
    REFUSE_REQUEST = "REFUSE_REQUEST"

@dataclass(frozen=True)
class History:
    step_index: int = 0
    retrieve_attempts: int = 0
    verify_attempts: int = 0
    ask_attempts: int = 0

@dataclass(frozen=True)
class Context:
    risk_tier: str = "high"
    max_steps: int = 4
    max_retrievals: int = 1
    max_verifications: int = 1
    max_asks: int = 2
    human_review_available: bool = True

def terminal_route(ctx):
    if ctx.human_review_available:
        return "REVIEW"
    return "ABSTAIN"

def evaluate_policy(record, ctx, history):
    hits = []

    def hit(name, priority, commitment, action,
            reasons=(), obligations=()):
        hits.append({
            "name": name,
            "priority": priority,
            "commitment": commitment,
            "action": action,
            "reasons": reasons,
            "obligations": obligations,
        })

    if record.get("policy_restricted") is True:
        hit("POLICY_PROHIBITED", 100,
            "DENY", "REFUSE_REQUEST",
            reasons=("POLICY_PROHIBITED",))

    if history.step_index >= ctx.max_steps:
        hit("MAX_STEPS_EXCEEDED", 95,
            "HOLD", terminal_route(ctx),
            reasons=("MAX_STEPS_EXCEEDED",))

    adequacy = record.get("epistemic_adequacy")

    if adequacy == "USER_GAP":
        if history.ask_attempts >= ctx.max_asks:
            hit("ASK_BUDGET_EXHAUSTED", 91,
                "HOLD", terminal_route(ctx),
                reasons=("USER_INPUT_MISSING",
                         "ASK_BUDGET_EXHAUSTED"))
        else:
            hit("USER_INPUT_REQUIRED", 90,
                "HOLD", "ASK",
                reasons=("USER_INPUT_MISSING",),
                obligations=("request_missing_user_field",))

    if adequacy == "RECOVERABLE_EVIDENCE_GAP":
        if history.retrieve_attempts >= ctx.max_retrievals:
            hit("RETRIEVAL_BUDGET_EXHAUSTED", 91,
                "HOLD", terminal_route(ctx),
                reasons=("EVIDENCE_GAP",
                         "RETRIEVAL_BUDGET_EXHAUSTED"))
        else:
            hit("RECOVERABLE_EVIDENCE_GAP", 90,
                "HOLD", "RETRIEVE",
                reasons=("EVIDENCE_GAP",),
                obligations=("retrieve_missing_evidence",))

    if ctx.risk_tier == "high" and record.get("provenance") in {
        "UNVERIFIED", "NOT_MEASURED", "UNAVAILABLE"
    }:
        if history.verify_attempts >= ctx.max_verifications:
            hit("VERIFICATION_BUDGET_EXHAUSTED", 86,
                "HOLD", terminal_route(ctx),
                reasons=("PROVENANCE_UNRESOLVED",
                         "VERIFICATION_BUDGET_EXHAUSTED"))
        else:
            hit("VERIFIED_PROVENANCE_REQUIRED", 85,
                "HOLD", "VERIFY",
                reasons=("PROVENANCE_UNRESOLVED",),
                obligations=("verify_provenance",))

    if record.get("sensitivity") == "FAIL":
        hit("SENSITIVITY_REVIEW", 70,
            "HOLD", "REVIEW",
            reasons=("SENSITIVITY_FAIL",),
            obligations=("review_context_dependence",))

    if not hits:
        required = {
            "containment": "PASS",
            "relation_fidelity": "PASS",
            "epistemic_adequacy": "ANSWERABLE",
            "provenance": "VERIFIED",
        }
        if all(record.get(k) == v for k, v in required.items()):
            hit("CLEAN_ACCEPT", 0, "PERMIT", "NONE",
                reasons=("ALL_REQUIRED_CONDITIONS_SATISFIED",))
        else:
            hit("UNHANDLED_OR_UNKNOWN_STATE", 60,
                "HOLD", terminal_route(ctx),
                reasons=("REQUIRED_STATE_NOT_SATISFIED",),
                obligations=("inspect_unhandled_state",))

    hits.sort(key=lambda x: x["priority"], reverse=True)
    winner = hits[0]

    reasons = list(dict.fromkeys(
        r for h in hits for r in h["reasons"]
    ))

    if winner["commitment"] == "DENY":
        obligations = list(winner["obligations"])
    else:
        obligations = list(dict.fromkeys(
            o for h in hits for o in h["obligations"]
        ))

    return {
        "commitment": winner["commitment"],
        "next_action": winner["action"],
        "matched_rule": winner["name"],
        "suppressed_rules": [h["name"] for h in hits[1:]],
        "reason_codes": reasons,
        "obligations": obligations,
        "step_index": history.step_index,
        "policy_version": "policy-v2",
    }

The implementation is intentionally small.

A production engine would add schema validation, policy profiles, explicit response modes, durable logs, and stronger rule composition.

But even this small engine demonstrates the chapter’s main claims.

For the tested examples it returned:

clean
→ PERMIT / NONE

provenance = UNVERIFIED
→ HOLD / VERIFY

policy_restricted = TRUE + sensitivity = FAIL
→ DENY / REFUSE_REQUEST
  matched = POLICY_PROHIBITED
  suppressed = SENSITIVITY_REVIEW

recoverable evidence gap, first attempt
→ HOLD / RETRIEVE

recoverable evidence gap, retrieval budget exhausted
→ HOLD / REVIEW

user information gap, clarification budget exhausted
→ HOLD / REVIEW

The important property is not that these routes are universal.

They are one explicit, testable profile.

12. Policy requires execution memory

The reliability loop is stateful.

Suppose policy returns:

RETRIEVE

The retriever finds nothing useful.

The system re-measures and sees the same evidence gap.

A stateless policy would return:

RETRIEVE

again.

And again.

And again.

The same problem exists for:

VERIFY
ASK
REFINE

So policy must see execution history:

step_index
retrieve_attempts
verify_attempts
ask_attempts
last_route
whether state improved

Then the policy can impose bounded recovery:

retrieve once
↓
no improvement
↓
REVIEW / ABSTAIN

or:

ask user twice
↓
required field still absent
↓
stop asking
↓
REVIEW / ABSTAIN

This second case is also a UX problem.

ASK is not a free backend action. It interrupts the user.

Repeated clarification can become clarification fatigue.

A mature router therefore has an ask budget just as it has retrieval and verification budgets.

In the reference run:

USER_GAP + ask_attempts = 0
→ ASK

USER_GAP + ask_attempts = 2
→ REVIEW

The general rule is:

A recovery route must have a stopping condition.

Without execution memory, a reliability loop can become an infinite retry loop.

13. Do not make another LLM the policy authority

A common architecture is:

generator LLM
    ↓
reliability record
    ↓
small router LLM
    ↓
action

The router may be useful as a proposal generator.

It should not be the deterministic policy floor.

If the same stochastic class of system is responsible for interpreting the rules that are supposed to constrain stochastic behavior, the control boundary becomes probabilistic again.

So distinguish:

LLM proposes route / drafts policy / explains diagnosis

from:

deterministic policy engine authorizes route

Learned models can absolutely contribute signals.

For example, an LLM may classify whether a source is likely authoritative.

But that output should enter the reliability record as a measurement with its own uncertainty and contract.

It should not silently become policy.

Open Policy Agent is a mature non-LLM example of the separation: structured application data is evaluated by a declarative policy decision point, while enforcement remains in the application.[4]

The model may propose. The policy engine decides. The enforcement point acts.

14. Policy as code means versioning, lineage, and replay

A policy decision should have durable decision provenance.

For example:

policy_trace = {
    "request_id": "req_8f29",
    "candidate_id": "cand_3",
    "generator": "generator-v17",
    "generator_output_hash": "sha256:...",
    "measurement_bundle": "measurements-v8",
    "measurement_schema_version": "measurement-schema-v4",
    "policy_input_schema_version": "policy-input-v3",
    "evidence_snapshot_uri": "immutable://evidence/7ca...",
    "evidence_snapshot_hash": "sha256:...",
    "policy_version": "policy-v2",
    "step_index": 1,
    "matched_rule": "VERIFIED_PROVENANCE_REQUIRED",
    "suppressed_rules": [],
    "commitment": "HOLD",
    "next_action": "VERIFY",
}

A hash proves identity.

A hash alone is not a replayable evidence snapshot.

The system also needs durable immutable content, or a content-addressed reference from which the exact evidence can be recovered.

This enables two different kinds of replay.

Policy-only replay

Freeze:

candidate
measurement record
evidence snapshot

and change only:

policy v1 → policy v2

This isolates the effect of policy.

Full decision replay

Freeze the original candidate and evidence, then rerun:

measurement stack v8 → v9
policy v2 → v3

This measures total evolution of the reliability stack.

Do not mix these experiments.

If a newer policy requires a field that an old measurement schema never produced, replay should not invent it.

Return something explicit such as:

POLICY_REPLAY_INDETERMINATE
missing_required_field = source_freshness

or re-measure from the preserved evidence snapshot.

15. Policy replay should produce a diff

The deliberately flawed opening policy provides a minimal replay example.

We evaluated five fixed records under:

policy-v1

which omitted provenance entirely — both the VERIFIED_PROVENANCE_REQUIRED rule in Section 11 and the provenance == VERIFIED line in its clean-accept check — and:

policy-v2

which added both back for the high-risk profile.

The deterministic replay produced (reproduced by experiments/hallucination/policy_engine.py):

That table is more useful than saying:

v2 is safer.

It tells us exactly which historical behavior changes.

A mature rollout sequence is:

    graph LR
    HPR[historical policy-only replay] --> LSP[live shadow policy]
    LSP --> CCD[compare counterfactual decisions]
    CCD --> CE[canary enforcement]
    CE --> SR[staged rollout]
  

A shadow policy reads live records and logs what it would have done without controlling execution.

Only after its disagreement set is understood should it receive enforcement authority.

16. Policy needs a set-valued oracle

A policy benchmark does not always have one uniquely correct route.

Take the provenance-gap record from Section 11. On the first pass the engine routes it to VERIFY. Replay the same record with verify_attempts already at the budget and it routes to REVIEW instead. Both are correct: both hold the commitment, both carry the resolve_provenance_before_commit obligation, and which one is right depends on execution history, not on the record alone.

A test whose expected value is the single string VERIFY now fails a correct system half the time — once the budget is spent. A test whose expected value is REVIEW fails it the other half. The bug is in the oracle, not the engine.

So policy labels should define constraints instead of one exact string.

For example:

oracle = {
    "must_not_commit": True,
    "required_obligations": [
        "resolve_provenance_before_commit",
    ],
    "acceptable_routes": [
        "VERIFY",
        "REVIEW",
    ],
}

This changes what we measure.

Useful policy metrics include:

Overall route accuracy can remain a convenience metric.

It should not be the primary safety metric.

Under-escalation and over-escalation have asymmetric costs.

And:

floor compliance
≠
policy adequacy.

A policy can perfectly obey an incorrect floor.

17. Progressive evaluation is an execution optimization, not an action ordering

A policy engine may run cheap checks first:

    graph TD
    HP[HARD PROHIBITIONS] --> CR[CHEAP REQUIRED FIELDS]
    CR --> PF[PROVENANCE / FRESHNESS]
    PF --> SC[STRUCTURAL / CONTAINMENT]
    SC --> CS[CONTEXT / SENSITIVITY]
  

This is an evaluation order.

It does not imply:

ASK < RETRIEVE < VERIFY < REVIEW

Those routes remain semantically different branches.

Progressive evaluation is useful for latency as well as cost.

Suppose a product has:

P99 latency budget = 200 ms

A clean candidate may pass cheap deterministic gates in 40 ms.

A high-risk candidate may require context perturbations that exceed the synchronous budget.

The correct response is not:

latency budget exceeded
→ silently ACCEPT

It is an explicit policy fallback such as:

HOLD
ASYNC_VERIFY
ABSTAIN
or
REVIEW

according to the product contract.

A reliability policy therefore owns not only epistemic risk but operational budgets.

18. Enforcement is a separate trust boundary

A prompt such as:

Only answer if every critical field is verified.

is guidance.

It is not enforcement.

The stronger architecture is:

GENERATOR
    ↓
MEASUREMENTS
    ↓
RELIABILITY RECORD
    ↓
POLICY ENGINE
    ↓
POLICY DECISION
    ↓
TRUSTED ENFORCEMENT POINT
    ↓
EXECUTION RECORD

Policy evaluation without trusted enforcement is advice.

A correct policy can still fail if the application ignores its output.

This gives four separate failure layers:

GENERATION FAILURE
MEASUREMENT FAILURE
POLICY FAILURE
ENFORCEMENT FAILURE

For example:

policy = DENY
execution = tool_call_succeeded

is not a hallucination failure.

It is an enforcement failure.

And:

Deterministic policy can be deterministically wrong.

That is why policy needs its own tests, oracles, replay, and review.

19. A minimum policy test suite

A serious policy suite should contain at least:

Clean commit cases

all required diagnostics satisfy contract
no prohibition

Expected:

PERMIT

Hard prohibition cases

Expected:

DENY + REFUSE_REQUEST

Recoverable evidence gaps

Expected constraint:

must_not_commit

acceptable route:

RETRIEVE

Missing user-owned variables

Expected:

HOLD + ASK

until the ask budget is exhausted.

Unknown critical measurements

Expected:

never implicit PERMIT

Conflicting rules

Example:

policy_restricted = TRUE
sensitivity = FAIL

Expected:

DENY + REFUSE_REQUEST
matched_rule = POLICY_PROHIBITED
suppressed_rules includes SENSITIVITY_REVIEW

Recovery-loop tests

RETRIEVE
→ no improvement
→ retrieve budget exhausted
→ terminal REVIEW / ABSTAIN

Floor property tests

For ordered escalation coordinates:

Replay schema tests

Historical record missing a newly required field:

→ POLICY_REPLAY_INDETERMINATE

not implicit pass.

20. Worked question: why are NOT_MEASURED, NOT_RUN, and NOT_APPLICABLE different?

Suppose a regulated reporting policy requires verified provenance.

Case A — NOT_MEASURED

The provenance sensor was expected to run but no result exists.

meaning:
required evidence about provenance is missing

route:
VERIFY / MEASURE / REVIEW

Case B — NOT_RUN

A hard legal prohibition already produced a terminal refusal before provenance measurement was needed.

meaning:
measurement intentionally skipped after route was determined

route:
keep terminal refusal

Case C — NOT_APPLICABLE

The candidate is a purely creative fictional passage for which provenance is outside the task contract.

meaning:
provenance has no semantic relevance

route:
ignore this field

Treating all three as None destroys precisely the information policy needs.

Typed missingness is part of the control architecture.

21. Using AI to help write policy

Section 13 said not to make an LLM the policy authority. This section is about letting an LLM help with policy authorship. Those are different: authorship produces candidate rules that are then translated to explicit semantics and tested; authority is the run-time decision, which stays deterministic.

An LLM can be useful during policy development.

Ask it to:

propose missing edge cases
generate synthetic policy records
suggest reason codes
translate prose requirements into candidate rules
review a replay diff

But do not give the generated policy enforcement authority merely because the prose looks plausible.

A useful adversarial exercise is:

Draft a reliability policy for a high-risk factual system.
Then test it on:

provenance = NOT_MEASURED
containment = PASS
epistemic_adequacy = ANSWERABLE

If the generated router effectively implements:

if state != "FAIL":
    allow()

it has failed the chapter’s most basic invariant.

The AI can help write the test.

The deterministic test decides whether the policy passes.

22. What you should now be able to answer

After this chapter, you should be able to explain:

1. Why raw measurement, diagnostic state, authorization, routing, and execution must remain separate.
2. Why a policy decision needs commitment, recovery, escalation, obligations, and reason codes rather than one flat action enum.
3. Why hard constraints should define the feasible action set before cost optimization occurs.
4. Why NOT_MEASURED, NOT_RUN, UNAVAILABLE, and NOT_APPLICABLE have different policy meanings.
5. Why policy floors apply only to genuinely ordered dimensions.
6. Why recovery routes require execution history and bounded retry budgets.
7. The difference between policy-only replay and full decision replay.
8. Why a correct policy without trusted enforcement is not a control.

23. Exercises

Exercise 1 — Separate policy dimensions

Rewrite this flat result:

action = VERIFY

as a typed decision containing:

commitment
next_action
escalation
response_mode
obligations
reason_codes

Explain what each dimension contributes.

Exercise 2 — Test unknown-state behavior

Create one critical field in each state:

PASS
FAIL
UNCERTAIN
NOT_MEASURED
NOT_RUN
UNAVAILABLE
NOT_APPLICABLE

Specify the allowed policy behavior for each.

Verify that only NOT_APPLICABLE can be ignored semantically and that NOT_RUN is justified by an earlier terminal route.

Exercise 3 — Test a policy floor

Define:

AUTO
REVIEW_REQUIRED
HUMAN_APPROVAL_REQUIRED

Generate random proposed and minimum escalations.

Assert:

Exercise 4 — Bound a recovery loop

Create a record with:

epistemic_adequacy = RECOVERABLE_EVIDENCE_GAP

Run policy before and after the retrieval budget is exhausted.

Expected:

first → RETRIEVE
after budget → REVIEW / ABSTAIN

Exercise 5 — Replay a policy change

Create five fixed measurement records.

Change one rule:

high-risk provenance UNVERIFIED

v1 → PERMIT
v2 → VERIFY

Produce a diff table and explain which behavior changed without regenerating any candidate.

Exercise 6 — Separate policy and enforcement failure

Construct:

policy commitment = DENY
execution status = COMPLETED

Identify the failed layer and the telemetry needed to prove it.

24. The deeper lesson

The first half of this book was largely about observation.

We learned:

fluency is not truth
retrieval is not support
support is not provenance
containment is not entailment
consistency is not correctness
sensitivity is not adequacy
confidence is not answerability

Now we can add:

Evidence is not authorization.

And we can make the statement stronger:

The model may propose.

The measurements diagnose.

The policy constrains.

The enforcement layer acts.

    graph TD
    GEN[GENERATOR] --> MEAS[MEASUREMENTS]
    MEAS --> REC[RELIABILITY RECORD]
    REC --> PD[POLICY DECISION]
    PD --> ENF[ENFORCEMENT]
    ENF --> ER[EXECUTION RECORD]
  

The policy decision becomes operational only when enforcement records what action was actually authorized and executed.

And because recovery is stateful, the loop can continue:

    graph LR
    M1[measurement] --> P1[policy] --> RET[RETRIEVE]
    RET --> NE[new evidence] --> RM[re-measure]
    RM --> P2[policy] --> VER[VERIFY]
    VER --> NE2[new evidence] --> RM2[re-measure]
    RM2 --> P3[policy] --> PER[PERMIT]
  

or terminate safely when recovery stops making progress.

This is the point where reliability stops being commentary on model behavior and becomes part of the software architecture.

Generation may be stochastic. Acceptance does not have to be.

Research roots

1. Chloe Autio, Reva Schwartz, Jesse Dunietz, Shomik Jain, Martin Stanley, Elham Tabassi, Patrick Hall and Kamie Roberts, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, 2024; NIST publication page updated in 2026. Provides voluntary lifecycle-oriented guidance for managing generative-AI risk in context rather than prescribing one universal threshold. https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence

2. Massimo Pacella, Gabriele Papadia and Vincenzo Giliberti, “Governed Agentic Process Automation: A Floor-Safety Guarantee for Compliance-Critical LLM Routing,” Algorithms 19(8), 627, 2026. Formalizes a closed routing space, policy-derived minimum escalation, deterministic fallback, and local floor-safety guarantee; evaluates the architecture on 166 onboarding profiles. https://www.mdpi.com/1999-4893/19/8/627

3. Aoun E. Muhammad, Kin-Choong Yow and Shrooq Alsenan, “Audit-as-code: a policy-as-code framework for continuous AI assurance,” Frontiers in Artificial Intelligence, 2026. Applies versioned machine-checkable governance gates, evidence bundles, risk-tier thresholds, and non-overridable blockers primarily at assurance / release scope. https://doi.org/10.3389/frai.2026.1759211

4. Open Policy Agent Project, “Open Policy Agent Documentation,” current documentation. Describes a general-purpose declarative policy engine that evaluates structured inputs separately from the application’s enforcement point. https://www.openpolicyagent.org/docs

Next: Verification, Repair, and Rejection

Chapter 12 can now express routes such as:

HOLD + RETRIEVE
HOLD + VERIFY
HOLD + REFINE
DENY + REJECT_CANDIDATE

But those are still instructions.

They need execution semantics.

If policy says:

VERIFY

what evidence should be gathered?

If it says:

REFINE

how do we repair a response without silently changing its meaning?

If a repair produces a new candidate, does the old policy decision still apply?

And when should the system stop trying and reject the candidate?

Chapter 13 turns policy routes into concrete recovery loops.

The next question is:

Once the system distrusts a candidate, how should it recover without creating a new failure?