The Investigation Engine

The first version of almost any verification tool is a single script. Here it would be run_math01: import SymPy, call Z3, shell out to lean, print the results. It works. Then someone wants a second claim, and copying the script duplicates the placeholder screening. A third claim needs Rocq; a fourth needs a notebook. Each copy decides afresh what “success” means, and sooner or later one of them reports a missing tool as a pass.

The repository still contains run_math01; it is now the “legacy runner”. When the generic runner replaced it, a parity check (part of the P2 phase) proved that the new runner produces canonically identical results for the same claims. That history is the argument of this chapter: the engine grew by extracting rules, and the acceptance test for each extraction was that the old answers did not change.

The engine is the layer that turns claims into replayable, hashed evidence bundles that fail closed. This chapter is a tour of its parts and of the design rules that hold them together. It draws on everything in Chapters 1 to 15.

The shape

 claim JSON  ──►  trusted adapters ──►  fail-closed registry
                                              │
                declarative case manifest ────┤
                                              ▼
                                      generic case runner
                                              │
                         EvidenceResult records + execution map
                                              │
                     validation / assessment / report (P1–P2)
                                              │
                          validation dossier (P3, no execution)
                                              │
                          reader-facing notebook (P4, no math)
                                              │
        ┌─────────────────┬───────────────────┼──────────────────┐
   external source     reconstruction       repair             external formalization
   (P5)                (P6)                 (P7)               (P8)

The labels P1 to P8 are the project’s phases, in the order they were built. Each later stage consumes the previous stage’s output.

Eight design rules

These rules recur in every part of the engine, and they are its real content.

1. Data declares; trusted code executes. A case manifest names an adapter by ID. It cannot supply a shell command, a module to import, an executable path, a working directory, an environment variable, source code or an arbitrary option. Manifest models reject unknown fields, so a command field fails validation. The worst a hostile manifest can do is choose among registered adapters.

2. Fail closed. The registry rejects duplicate IDs, unknown adapters (before probing), a probe key shared across systems, and any adapter whose system, family, purpose or supported claims do not match the check. The registry is frozen before a run. Being registered never counts as evidence.

3. Availability is not success. A missing executable yields not_installed / not_evaluated for every check that needed it, with no mathematical conclusion. A timeout is inconclusive, timed_out, with no retries. External-proof timeouts are set by the engine and capped at 600 seconds; a manifest that asks for more fails validation.

4. Keep the failure. Failures are results, not swallowed exceptions. Placeholder rejections, kernel rejections, timeouts and absent tools all appear as typed records with their reasons. At the project level, the failed CI runs of Chapter 8 stay on the record.

5. Execute once, project many times. Each stage after execution consumes the previous stage’s bundle and runs nothing mathematical. P3 builds dossiers from a P2 bundle with zero adapter executions. P4 renders a notebook from a P3 bundle and executes only two trusted Python cells. Later phases count their mathematical executions explicitly (P6: six; P7: eleven new; P8: one build, one axiom audit, one Comparator run) and show zero for their dossier, report and notebook steps.

6. Check by regeneration, not by trust. Each bundle has an owner manifest and a verifier that rebuilds the derived parts and compares them byte for byte: the companion report, the index, the notebook source. Verification also replays the notebook and compares its canonical hash. A tampered or incomplete bundle fails. The committed plan, not the bundle’s own snapshot, is the authority for verification.

7. Staged publication that fails closed, but is not atomic. Bundles are staged, verified, then copied in an order that writes the index last and the owner last, with cleanup on failure. The P4 record is explicit that this is not an atomic directory swap, and the engine never claims otherwise.

8. Accepted evidence does not change. Protected output directories are refused. Every phase from P4 on restates the SHA-256 of the accepted MATH01 report to show it did not move.

The parts

One claim, start to finish

Take the algebraic Lorentz claim.

1. Claim. lorentz-algebraic.claim.json, fingerprint 5493a872… (Chapter 1).
2. Manifest. math01.case.json pins the claim version and fingerprint and lists its checks: Z3, Lean, Rocq and Isabelle (advisory).
3. Run. case run probes each tool once, executes each check, screens for placeholders and records the results: Z3 unsat, Lean and Rocq accepted, Isabelle not_installed.
4. Assess. assess_claim applies the precedence rules of Chapter 6 to the results whose claim ID, version and fingerprint all match.
5. Dossier. P3 reconciles every check with its result, runs structural correspondence and computes the bounded conclusion: source_claim_qualified on the hosted runner, incomplete locally without Rocq.
6. Notebook. P4 displays the conclusion, verified against P3, and fails if it differs.

At no step does any layer decide what a lower layer meant.

What this establishes

The engine’s value is that every sentence in a result can be traced to a record with a hash, a tool version and a stated limit. Its cost is size: the machinery for saying “we do not know” is much larger than the machinery for computing. As of main after P8 (counted 2026-10-27), there were about 35,800 lines of Python under src/ and 31 test files, and the P8 record’s full suite passed 400 tests with 1 skipped. Most of that is the bundle machinery of P3 to P8; the original MATH01 core is a small fraction.

The engine does not assume that its adapters agree; it reports their agreement at the level Chapter 15 defines. What it cannot do is decide whether a claim is worth investigating, or whether a claim contract captures the mathematician’s intent. Those remain human judgments (Chapter 7). The design rules above can be checked by reading the registry and runner code; no independent code review of them is recorded.

Try it yourself

uv run math-engine adapters list
uv run math-engine case validate experiments/relativity/math01.case.json
uv run math-engine case run experiments/relativity/math01.case.json \
  --clean --output .ci/math-val01-p2
uv run python scripts/ci_math_val01_p2_parity.py

The first two commands print the adapter registry and validate the case manifest (executed):

$ python -m math_engine adapters list
[ { "adapter_id": "isabelle.lorentz-algebraic.v1",
    "dependency_family": "isabelle_kernel",
    "produces_evidence": true, "supported_purposes": ["proof"],
    "system": "isabelle", ... },
  { "adapter_id": "lean4.gamma-bridge.v1", ... }, ... ]

$ python -m math_engine case validate experiments/relativity/math01.case.json
{ "claims": 4, "checks": 10, "capability_probes": 1,
  "binding_audit": "declared_metadata_only",
  "manifest_id": "case-manifest-bff89029d60e…", ... }

The registry lists 28 adapter IDs in a fixed order: the eleven original MATH01 operations plus the P6 and P7 additions (*.p6-*, *.p7-*), each with its system, dependency family, purposes and the claims it supports. The manifest validates as four claims and ten checks. “binding_audit: declared_metadata_only” is the L0 level of Chapter 15, written into the manifest itself.

Seeded error. Add a "command": ["echo", "hi"] field to a check in a copy of the manifest. Executed on 2026-11-09 with ch16_invalid_manifest.py, validation rejects the copy:

error: case manifest failed validation: 1 validation error for CaseManifest
claims.0.checks.0.check.command
  Extra inputs are not permitted [type=extra_forbidden, input_value=['echo', 'hi'], input_type=list]
rejected unknown command field: True

Open questions.

1. The pipeline from P3 to P8 repeats one pattern: plan, build, verify, check consistency. Is there now enough duplication to justify a generic bundle framework, or would it hide the differences between phases that carry the safety rules?
2. Rule 6 trusts the verifier code to rebuild honestly. What verifies the verifier, and is the test suite playing that role?
3. How much of the engine’s size is the cost of the “no confidence score” rule, and would a reader accept a smaller engine that said less?

Sources

No software-architecture literature on fail-closed design, provenance or reproducible builds is cited. In the repository: docs/architecture/overview.md, adapter-registry.md, theorem-communication.md, validation-dossier-pipeline.md and investigation-notebook.md, and the P2 to P8 evidence records.