Example

Break Euler's factorization on purpose

Mutate the divisor, cofactor and index, require each rejection at the changed line, and catch a control that passes by crashing.

A check that has never rejected anything has not shown that it can. This program runs the chapter’s experiment on a tiny checker: three mutations of Euler’s factorization must each be rejected, at the line that was changed. Then it shows how a control can reassure falsely: a checker that crashes still “fails”, and a control that only expects failure passes.

Run it

Save this as failure_controls.py and run it with Python 3.11 or later. It uses the standard library only.

"""Failure controls: a check that cannot fail proves nothing (standard library only).

The 'proof file' is a list of lines; the checker evaluates each claimed equation exactly,
like Lean's `decide`, and reports the line that failed.
"""
PROOF = [
    "# Euler's witness against Fermat",
    "2 ** (2 ** 5) + 1 == 641 * 6700417",
    "(2 ** (2 ** 5) + 1) % 641 == 0",
]


def check(lines):
    for number, line in enumerate(lines, 1):
        if line.startswith("#"):
            continue
        if not eval(line, {"__builtins__": {}}):
            return f"line {number}: proposition is false: {line}"
    return "accepted"


def mutate(old, new):
    return [line.replace(old, new) for line in PROOF]


print(f"the proof:            {check(PROOF)}")
CONTROLS = {
    "wrong-divisor 643": (mutate("641", "643"), "line 2"),
    "wrong-cofactor":     (mutate("6700417", "6700419"), "line 2"),
    "wrong-index n=4":    (mutate("2 ** 5", "2 ** 4"), "line 2"),
}
print("\ncontrols must be rejected, at the line that was changed:")
for name, (lines, where) in CONTROLS.items():
    verdict = check(lines)
    right_reason = verdict.startswith(where)
    print(f"  {name:18} right place: {right_reason!s:5} {verdict}")


def run_control(lines, checker, expect_text=None):
    """A control 'passes' if the run fails (and, when given, says the expected thing)."""
    try:
        verdict = checker(lines)
    except Exception as e:
        verdict = f"crashed: {type(e).__name__}: {e}"
    failed = verdict != "accepted"
    ok = failed and (expect_text is None or expect_text in verdict)
    return ("passed" if ok else "FAILED"), verdict


def chekc(lines):                                    # a mistyped checker: it can never run
    return chekc_impl(lines)                         # noqa: F821


print("\na control that reassures falsely (the checker is misspelt, so it crashes):")
print("  expecting failure only:      ", run_control(CONTROLS["wrong-divisor 643"][0], chekc))
print("  expecting 'proposition is false':", run_control(CONTROLS["wrong-divisor 643"][0], chekc, "proposition is false"))
print("\nand the real checker under the strict control:")
print("  ", run_control(CONTROLS["wrong-divisor 643"][0], check, "proposition is false"))

Expected outcome

the proof:            accepted

controls must be rejected, at the line that was changed:
  wrong-divisor 643  right place: True  line 2: proposition is false: 2 ** (2 ** 5) + 1 == 643 * 6700417
  wrong-cofactor     right place: True  line 2: proposition is false: 2 ** (2 ** 5) + 1 == 641 * 6700419
  wrong-index n=4    right place: True  line 2: proposition is false: 2 ** (2 ** 4) + 1 == 641 * 6700417

a control that reassures falsely (the checker is misspelt, so it crashes):
  expecting failure only:       ('passed', "crashed: NameError: name 'chekc_impl' is not defined")
  expecting 'proposition is false': ('FAILED', "crashed: NameError: name 'chekc_impl' is not defined")

and the real checker under the strict control:
   ('passed', 'line 2: proposition is false: 2 ** (2 ** 5) + 1 == 643 * 6700417')

Each mutation is rejected on line 2, the line that was changed: failing for the right reason. The misspelt checker crashes with a NameError; the control that expects only “not accepted” counts that as a pass, and tests nothing. The same control insisting on the verdict text catches it. The laboratory’s controls require both the failure and the expected message.

Mechanism and limitations

Documented: Chapter 23 of the book sets out the mathematics; this program is a self-contained demonstration of it, not one of the laboratory’s recorded investigations.

Observed: the output above, produced by running the program exactly as shown.

Not shown:

  • The checker evaluates Python expressions; it stands in for Lean’s decide on the same equations.
  • Passing controls show the check can fail in these ways, not that it catches every error.

Related worked example: Chapter 2 , Chapter 7 .

Challenge this result

Copy this into the AI assistant of your choice, or work through it yourself. Check any finding against the recorded evidence.

Examine the three mutations and the defective control. For each mutation,
say what it tests and what a check that ignored it would have to be doing.
Write a fourth mutation that the checker would wrongly accept, and propose a
control that would catch it. Then explain why a control must specify how it
fails, not only that it fails.

Chapter: https://programmer.ie/books/math/23-chapter/
Example: https://programmer.ie/examples/math/23-chapter/
Program: https://github.com/ernanhughes/math/blob/main/notebooks/math/programs/23-failure-controls.py
Distinguish what the record shows from what you infer. If you cannot open a link, ask me for the material; do not invent it.

Canonical source and the program .

Understand this example

Copy this prompt into your AI tool. No code runs here.

Apply this example

Copy this prompt into your AI tool. No code runs here.