Break Euler's factorization on purpose
Mutate the divisor, cofactor and index, require each rejection at the changed line, and catch a control that passes by crashing.
A check that has never rejected anything has not shown that it can. This program runs the chapter’s experiment on a tiny checker: three mutations of Euler’s factorization must each be rejected, at the line that was changed. Then it shows how a control can reassure falsely: a checker that crashes still “fails”, and a control that only expects failure passes.
Run it
Save this as failure_controls.py and run it with Python 3.11 or later. It uses the standard library only.
"""Failure controls: a check that cannot fail proves nothing (standard library only).
The 'proof file' is a list of lines; the checker evaluates each claimed equation exactly,
like Lean's `decide`, and reports the line that failed.
"""
PROOF = [
"# Euler's witness against Fermat",
"2 ** (2 ** 5) + 1 == 641 * 6700417",
"(2 ** (2 ** 5) + 1) % 641 == 0",
]
def check(lines):
for number, line in enumerate(lines, 1):
if line.startswith("#"):
continue
if not eval(line, {"__builtins__": {}}):
return f"line {number}: proposition is false: {line}"
return "accepted"
def mutate(old, new):
return [line.replace(old, new) for line in PROOF]
print(f"the proof: {check(PROOF)}")
CONTROLS = {
"wrong-divisor 643": (mutate("641", "643"), "line 2"),
"wrong-cofactor": (mutate("6700417", "6700419"), "line 2"),
"wrong-index n=4": (mutate("2 ** 5", "2 ** 4"), "line 2"),
}
print("\ncontrols must be rejected, at the line that was changed:")
for name, (lines, where) in CONTROLS.items():
verdict = check(lines)
right_reason = verdict.startswith(where)
print(f" {name:18} right place: {right_reason!s:5} {verdict}")
def run_control(lines, checker, expect_text=None):
"""A control 'passes' if the run fails (and, when given, says the expected thing)."""
try:
verdict = checker(lines)
except Exception as e:
verdict = f"crashed: {type(e).__name__}: {e}"
failed = verdict != "accepted"
ok = failed and (expect_text is None or expect_text in verdict)
return ("passed" if ok else "FAILED"), verdict
def chekc(lines): # a mistyped checker: it can never run
return chekc_impl(lines) # noqa: F821
print("\na control that reassures falsely (the checker is misspelt, so it crashes):")
print(" expecting failure only: ", run_control(CONTROLS["wrong-divisor 643"][0], chekc))
print(" expecting 'proposition is false':", run_control(CONTROLS["wrong-divisor 643"][0], chekc, "proposition is false"))
print("\nand the real checker under the strict control:")
print(" ", run_control(CONTROLS["wrong-divisor 643"][0], check, "proposition is false"))Expected outcome
the proof: accepted
controls must be rejected, at the line that was changed:
wrong-divisor 643 right place: True line 2: proposition is false: 2 ** (2 ** 5) + 1 == 643 * 6700417
wrong-cofactor right place: True line 2: proposition is false: 2 ** (2 ** 5) + 1 == 641 * 6700419
wrong-index n=4 right place: True line 2: proposition is false: 2 ** (2 ** 4) + 1 == 641 * 6700417
a control that reassures falsely (the checker is misspelt, so it crashes):
expecting failure only: ('passed', "crashed: NameError: name 'chekc_impl' is not defined")
expecting 'proposition is false': ('FAILED', "crashed: NameError: name 'chekc_impl' is not defined")
and the real checker under the strict control:
('passed', 'line 2: proposition is false: 2 ** (2 ** 5) + 1 == 643 * 6700417')Each mutation is rejected on line 2, the line that was changed: failing for
the right reason. The misspelt checker crashes with a NameError; the control
that expects only “not accepted” counts that as a pass, and tests nothing. The
same control insisting on the verdict text catches it. The laboratory’s
controls require both the failure and the expected message.
Mechanism and limitations
Documented: Chapter 23 of the book sets out the mathematics; this program is a self-contained demonstration of it, not one of the laboratory’s recorded investigations.
Observed: the output above, produced by running the program exactly as shown.
Not shown:
- The checker evaluates Python expressions; it stands in for Lean’s
decideon the same equations. - Passing controls show the check can fail in these ways, not that it catches every error.
Related worked example: Chapter 2 , Chapter 7 .
Challenge this result
Copy this into the AI assistant of your choice, or work through it yourself. Check any finding against the recorded evidence.
Examine the three mutations and the defective control. For each mutation,
say what it tests and what a check that ignored it would have to be doing.
Write a fourth mutation that the checker would wrongly accept, and propose a
control that would catch it. Then explain why a control must specify how it
fails, not only that it fails.
Chapter: https://programmer.ie/books/math/23-chapter/
Example: https://programmer.ie/examples/math/23-chapter/
Program: https://github.com/ernanhughes/math/blob/main/notebooks/math/programs/23-failure-controls.py
Distinguish what the record shows from what you infer. If you cannot open a link, ask me for the material; do not invent it.Canonical source and the program .
Understand this example
Copy this prompt into your AI tool. No code runs here.
Apply this example
Copy this prompt into your AI tool. No code runs here.