Example

Bind a check to the state it actually reached

Run one declared check over four situations and see PASS, FAIL, INCONCLUSIVE and ERROR kept apart, with the exit-code mapping declared before the command runs.

A check that passes establishes one narrow thing: this declared command, against this bound target, exited 0. The chapter’s claim is that the interesting cases are the other three, and that INCONCLUSIVE and ERROR are not FAIL. A check that could not read its subject, and a check that measured the wrong state, need different repairs from a check that found a real problem.

Run it

git clone https://github.com/ernanhughes/codeai.git
cd codeai
git checkout c1707f793abbd6042dce7d00b6dde34f2dbf33cc
PYTHONPATH=src python examples/applied_ai/ch21_verification.py

CodeAI has no runtime dependencies, so nothing needs installing. Python 3.11 or later. No model is called and nothing leaves the machine.

Expected outcome

Four situations, four verdicts, and three different repairs:

PASS expected     : PASS         binding=bound       -> accept the work
                    exit 0
FAIL expected     : FAIL         binding=bound       -> investigate the work
                    exit 1
cannot tell       : INCONCLUSIVE binding=bound       -> gather better evidence
                    exit code 2 is declared inconclusive by exit-code-v1: binary content: this check cannot read it as prose
target moved      : ERROR        binding=mismatch    -> repair the measurement

The ERROR case is the interesting one, and the run shows why: the target’s state hash no longer matched the one the request named, so nothing was measured against the state that was requested.

And the run records which checks actually reached a verifier:

verifier ran for the inconclusive check : True
verifier ran for the errored check      : False
recorded policy for k-maybe             : exit-code-v1 inconclusive=[2]

The binding, and the exit-code policy

The reduction a chapter can print:

    binding = bind(target)
    if not binding.permits_verification:
        return ERROR                  # nothing was measured
    result = verifier.run(request)

    FAIL         -> investigate the work
    INCONCLUSIVE -> gather better evidence
    ERROR        -> repair the measurement

The exit-code mapping is declared before the command runs and recorded with the result. It is a statement about this command, not a convention baked into every command that ever exits 2:

class ExitCodePolicy:
    """How a command's exit code maps to a verdict, declared before it runs.

    Declaring ``2`` inconclusive for one check is a statement about that
    command. Baking it into the runtime would smuggle a convention into every
    command that ever exits 2, which is exactly the collapse this seam refuses.

    ``fail_codes=None`` keeps the ordinary shell reading: anything not declared
    otherwise is a failure. Declaring ``fail_codes`` explicitly makes the policy
    total, and an exit code outside it is then an unmapped outcome: ERROR, not a
    guess.
    """

    policy_id: str = "exit-code-v1"
    pass_codes: tuple[int, ...] = (0,)
    fail_codes: tuple[int, ...] | None = None
    inconclusive_codes: tuple[int, ...] = ()

    def verdict_for(self, exit_code: int) -> str | None:
        """The declared verdict, or None when the code is unmapped (an ERROR)."""
        if exit_code in self.pass_codes:
            return CheckVerdict.PASS
        if exit_code in self.inconclusive_codes:
            return CheckVerdict.INCONCLUSIVE
        if self.fail_codes is None or exit_code in self.fail_codes:
            return CheckVerdict.FAIL
        return None

Mechanism and limitations

Documented: the reduction and the four outcomes are in the program’s docstring in examples/applied_ai/ch21_verification.py; ExitCodePolicy is in src/codeai/verification.py; the runtime’s own check path is Runtime.run_check in src/codeai/runtime.py.

Observed: the run above. The exit-code policy mapped 0 to PASS, 1 to FAIL and 2 to INCONCLUSIVE, and the policy was recorded with the result. The moved target produced ERROR with a binding mismatch, and the verifier did not run at all for that case — the run reports that explicitly.

Not shown. The check here is a marker in a file. A PASS says the marker was gone and the declared command could see that; it says nothing about whether the criterion was the right one, or whether the work was good. The INCONCLUSIVE case is produced by making the file binary, which is a specific way for a check to fail to read its subject, not a general claim about inadequacy. The binding here is a state hash of one target; a target that changes for a reason the check does not model would produce the same mismatch. Nothing here measures whether a verification verdict improved a decision.

Canonical source and the CodeAI verification module .

Understand this example

Copy this prompt into your AI tool. No code runs here.

Apply this example

Copy this prompt into your AI tool. No code runs here.