Independent Reproduction

“Reproduced” can mean at least six different things:

• I ran it again and got the same output.
• A fresh build on my machine produced the same hashes.
• A CI server I do not control ran it and it passed.
• Someone else downloaded the bundle and verified it.
• Someone else, with different tools, re-derived the result.
• Someone else wrote their own implementation from the specification.

The first is a weak check and the last a strong one, and they are not interchangeable. A report that uses the word without saying which is making a stronger claim than its author may realize.

This chapter gives a graded vocabulary for “has this been reproduced?”, with the project’s actual standing at each level. It extends Chapter 8 from mechanics to evidence, and builds on the kernel chapters (Chapters 11 and 12) and the cross-system ladder (Chapter 15).

The ladder

The reproducibility ladder. More independence answers a different question; it does not by itself establish mathematical truth, and the project has not reached R4.

The evidence, level by level

R1: clean rebuilds

In the P4 record, two clean builds in one environment produced identical source notebook bytes, executed bytes, canonical executed hash, execution-record semantic ID and index semantic ID. The full index hash differed, by design, because it covers the generation time (Chapter 8). The same pattern holds for P6, P7 and P8.

This shows that the bundle is deterministic, given its inputs. It does not show that the inputs are right, or that a different environment would give the same result.

R2: hosted runs, and their failures

The R2 hosted qualification needed seven runs. The first six failures were integration defects (build order, lost logs, Mathlib cache extraction on Windows), and none weakened a control (Chapter 8). Later failures were kept in the same way: an R4 run that failed because PhysicalComposition.vo was not yet built, and the first P8 hosted attempt, which ran out of disk space.

A hosted pass means something because the runner is not the author’s machine, it uses a pinned toolchain (elan-pinned Lean 4.35.0, Rocq 9.0.1 from opam), and the workflow checks that uv.lock and lake-manifest.json are unchanged. It does not show that a different Ubuntu image, a different mirror or a different year would agree.

R3: downloading what was published

In the P6 baseline for P7, the post-merge artifact was downloaded, “its SHA-256 matched the GitHub digest,” and the downloaded bundle “independently passed P6 verification and consistency checks and reproduced” the three conclusions, the index and the canonical notebook hash. A third P7 build from the downloaded P6 bundle produced the same P7 index.

This is the strongest reproduction in the repository, but it was performed by the same project, on the same kind of machine as development. It shows that the published artifact is what the pipeline says it is. It is not independent.

R4: the missing level

The statuses are explicit. In P5’s review set, independent_reproduction is not_evaluated. The R1 record concedes that the first run “was executed in a prepared Windows worktree, not from an empty machine.” Rocq, Isabelle and SageMath have been run on the drafting machine only in Docker, outside the engine (Chapters 12 to 14). A reader who wants R4 can follow docs/methodology/reproducibility.md and report back. A record of that report would move the dimension from not_evaluated to a real value.

R5: independent implementations

There are three kinds, all partial:

• Two kernels (Lean and Rocq) check separately written proofs of statements that the R3 audit aligns structurally (L1) and that share an interface (L2). The real-number models are not shown to be equivalent (L3; Chapters 12 and 15).
• Two computational tools (SymPy and Z3) agree on residuals and witnesses, although both were driven by the same author’s encoding (Chapters 9 and 10).
• A Comparator (P8) is an independent checker of an external Lean project, and it runs on the same kernel (Chapter 28).

None is an independent reimplementation of the engine, and none is a different team’s formalization of the same claims.

What more kernels add

A second kernel guards against a proof-checker bug, a library quirk, or a Lean-specific convention doing hidden work. It does not guard against a shared misstatement. The Pythagorean case is typical: Lean and Rocq accept the coordinate theorem, the statements are structurally_aligned, and the historical Euclid claim remains not_evaluated (Chapter 25). More kernels would not change that. A person comparing the Greek text with the formalization would.

External review

The P5 conventions list twelve review dimensions, including independent_reproduction, statement_correspondence and mathematical_argument, and require each to say what a person actually inspected. Today every dimension that depends on a person is not_evaluated for the project’s own claims. P8 shows what a partial case looks like for an external claim: a human comment judged plausible_not_checked, recorded, but not an approval (Chapter 28). The book’s own chapters are in the same position: drafted with AI assistance, with no human review recorded (Chapter 7).

What this establishes

The honest summary: the project’s results are well reproduced at the levels it controls (R0 to R3) and unreproduced at the levels that give outside readers independence (R4, and R5 for the engine). That is a statement about how mature the process is, not about the mathematics, which is elementary and which the kernels check.

Try it yourself

This chapter asks you to do R4. On a machine the project has never touched, with Lean 4.35.0, Mathlib v4.35.0 and Rocq 9.0.1 installed according to the reproducibility guide, run

uv sync --all-groups --locked
uv run math-engine math01 run --clean --output artifacts/math01
uv run python scripts/ci_report_consistency.py --validate-only artifacts/math01/report.json

and report what differs. A difference is a result.

Open questions.

1. Is R3 really “partial”, or already “yes”? It depends on whether a machine the author did not configure counts. The author did not configure the hosted runners, but did run the downloaded-artifact check.
2. Which of Lean 4.35.0, Mathlib v4.35.0 or the Ubuntu image would most likely drift first, and would any recorded check notice?
3. What is the smallest R4 the project could arrange, and who is the right independent party?

Evidence record

R1 identities for P6: index p6-qualification-index-5f529257… and canonical hash d406f924…, the same in two builds. Hosted runs are listed in Chapter 8.

Sources

No literature on reproducibility is cited. In the repository: docs/methodology/reproducibility.md, docs/research/MATH01-status.md, docs/research/MATH-VAL01-P4-evidence.md, docs/research/MATH-VAL01-P6-evidence.md, docs/research/MATH-VAL01-P7-evidence.md, docs/research/MATH-VAL01-P8-evidence.md and content/planning/external-source-review.md.