Part II — Debugging Deterministic Software

The corruption nobody heard

Chapters 5–7 kept finding breaks at the same place: the handoff. Now watch what happens when the handoff has no guard.

The clean → summarize pipeline from Chapter 4 runs green for a month. Then finance notices refund totals drifting downward — no crash, no traceback, no red badge. Reconstruction shows the clean refactor (the one Chapter 4 convicted) had been dropping refund_id for split-shipment children for four weeks. Chapter 5’s .get() silence would have done the same even faster: every affected row computed a total without its refund, wrote it to the report, and moved on. The program never failed. It just lied, quietly, 12,000 rows per night.

OBSERVATION: report totals for split-shipment orders are lower than ledger truth by exactly the refund amounts, across four weeks of runs, with zero exceptions raised on any run. HYPOTHESIS H1 (late corruption): summarize miscomputes totals it was given good rows for. HYPOTHESIS H2 (silent handoff violation): clean emits rows violating the downstream contract, and summarize propagates them without complaint. INFERENCE: none yet — the report symptom is downstream of both; only a handoff check separates them.

This chapter’s question: how do you convert what Chapters 5–7 found into an artifact that fails loudly the next time, at the handoff, on the first bad row?

Why detection without prevention fails

Three habits let a diagnosed defect return:

1. Fixing the instance, not the contract. The engineer restores refund_id propagation in clean and closes the ticket. The knowledge — “summarize requires refund_id on every row” — lives in the ticket, the chat log, and one engineer’s memory. The next refactor, six months later, drops a different key through the same unguarded handoff.
2. Tests that assert outputs, not contracts. The suite checks summarize(fixture) == expected_report. That test passes on the fixture and says nothing about the handoff rule. A new fixture shape that violates the contract produces a new wrong report that the output test never names as a contract break — it just fails opaquely, or worse, passes because the fixture never covers the shape.
3. Fear of asserts in production. “Asserts get optimized away with -O; validation is expensive; we’ll log instead.” So the handoff gets a log line nobody alerts on, or a .get() default that converts the next violation into the next silent drift. The objection confuses mechanism with intent: the point is a loud, local failure at the handoff, whether implemented as assert, explicit raise, or schema validation. Pick the mechanism that actually fires in your runtime — but fire something.

The mental model: every handoff Chapters 5–7 convicted implies a contract, and every contract belongs in the code, not in the ticket. A contract states what the producer guarantees and what the consumer requires, in executable form, at the boundary line. An assertion is a contract with a tripwire: when the next violation arrives, the program stops at the handoff with the producer’s name on the failure — instead of writing 12,000 wrong rows.

The method: write the contract the failure taught you

Contracts come in three strengths. Use the weakest that still fails loudly:

This vocabulary is not new. Meyer’s Design by Contract, built into Eiffel, formalized exactly this split: a routine has a precondition the caller must satisfy and a postcondition the routine guarantees in return, and a class carries invariants true between calls (Meyer, 1992). Meyer’s attribution rule is the one this chapter uses to name the offender: a violated precondition is the caller’s bug, a violated postcondition is the routine’s bug. A consumer-entry guard that fires is asserting “my precondition was broken” — and it points upstream by construction.

The conversion procedure, applied to the clean → summarize handoff:

# billing/totals.py — the contract the incident taught (constructed illustration)
REQUIRED_KEYS = ("order_id", "subtotal", "refund_id")

def summarize(orders):
    # Handoff contract: every row entering summarize carries the full schema.
    # Placed HERE (consumer entry), so the failure names the handoff, not line 88.
    bad = [r for r in orders if not all(k in r for k in REQUIRED_KEYS)]
    if bad:
        raise ValueError(
            f"summarize contract violated: {len(bad)} rows missing keys; "
            f"first offender keys={sorted(bad[0].keys())}"
        )
    return [_row_total(o) for o in orders]

Placement is the decision most engineers get wrong. The check goes at consumer entry (first line of summarize), not at the crash line (line 88) and not solely at producer exit. Rationale: consumer entry is where the requirement is known; a producer-exit check rots when a second producer appears. Ideally both — but if one, consumer entry, because that is where the next unknown producer’s output will arrive.

    flowchart LR
    P1["clean (producer)"] --> G
    P2["a future second producer"] -.->|later| G
    G["contract at summarize entry: every crossing row carries the schema"] --> S["summarize (consumer)"]
    G -.->|violation| R["raise — names the handoff and first offender, before line 88 runs"]
  

Cost honesty: the guard scans keys per row — O(rows × keys), trivial against I/O-bound pipeline stages. Where the handoff is hot (inner loops, per-token paths), hoist the check to batch entry or sample it — but write down that you sampled, so nobody mistakes a sampled guard for a complete one.

FORECAST (testable prediction, not a result): with this guard in place, the Chapter 4 refactor that dropped refund_id fails on its first test run with ValueError: summarize contract violated, naming the handoff — instead of shipping four weeks of quiet drift. UNKNOWN: whether any historical reports beyond the four-week window are affected — the guard prevents recurrence; it does not audit the past.

Demonstration: the assertion that would have paged at 02:14

Replay the Chapter 5 incident with the contract installed. Night of the refactor: the job runs, clean emits its first refund_id-less child row, summarize raises ValueError on row one, the traceback names the handoff (totals.py consumer entry, first offender keys listed), and the on-call engineer sees a contract violation — not a KeyError at line 88 inviting a .get() silence. Diagnosis starts convicted at the handoff instead of starting from a misleading last line.

Pair it with Chapter 7’s seam: the chunk_count fencepost also earns a contract — assert result == math.ceil(n / size) if n else 0 as a cross-check, or a postcondition assert all(len(p) <= size for p in pages) and sum(map(len, pages)) == n. Boundary rules that cannot state their postcondition are rules nobody understood well enough to keep.

What the contract does not do: it does not prove summarize computes correct totals (H1 in this chapter’s pair needs its own output tests), and it does not name which upstream change broke the producer. It converts H2-class silent propagation into a loud, local, attributable stop. That is the whole job, and it is enough.

Research lineage: contracts earn their cost

Assertions correlate with fewer faults, in commercial code. Kudrjavets, Nagappan, and Ball studied two Microsoft components whose developers used assertions systematically and found that as assertion density in a file rose, fault density fell by a statistically significant amount; the assertions also caught a large share of the bugs later recorded in the defect database (Kudrjavets, Nagappan & Ball, 2006). This is the evidence behind “fire something at the handoff” — the objection that guards are cost without benefit does not survive contact with the data.

The contract the failure taught you can be partly inferred. Daikon watches a program’s runs and reports the properties that held across all of them — equalities, ranges, non-null fields, ordering — as likely invariants (Ernst et al., 2001). Point it at the passing runs from before the incident and it will surface “refund_id present on every row entering summarize” as a candidate contract. The catch, which the Daikon authors report themselves, is noise: it also proposes many invariants that are coincidental or useless, so the output is a candidate list to curate, not a contract to install wholesale. The incident tells you which candidate matters.

Heavy assertion use is standard in safety-critical code. JPL’s “Power of Ten” rules for spaceflight software mandate a minimum assertion density and require assertions to be side-effect-free and to trigger an explicit recovery action, not just abort. Money-path handoffs are closer to that regime than to “we’ll log it.”

This apparatus does not stop at deterministic code. Chapter 46 builds the same precondition/postcondition/invariant contracts around an AI debugging assistant — the report must ground every observation, the plan must be plural — with an independent checker in the DbC role. Chapter 54 makes the leap explicit: a runtime guardrail on a model’s output is an assertion whose predicate is a learned function rather than a key check — the “model assertion” (Kang et al.). Daikon-style inference of “which invariant held across the passing runs” reappears there too, as a way to propose guardrail clauses from pre-incident traffic. The move is identical; only the predicate gets harder to write.

Lab 8: add the contract, watch it fire

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own firing record.

Setup. Take the pinned repro from any of Chapters 5–7 (the refund_id handoff, the None-coupon caller boundary, or the chunk_count seam). You need two runs: the currently-passing-but-shouldn’t state (silent corruption or unguarded handoff) and the post-contract state.

Task.

1. Write the handoff contract in one sentence before coding it: “Every ___ crossing ___ guarantees ___.” Name producer, consumer, and the exact predicate.
2. Implement the weakest loud mechanism for your runtime (assert for dev invariants; explicit raise for money paths — justify the choice in one line).
3. Run the pinned violating input. Independent variable: contract absent vs. present. Controlled variables: input, code version apart from the added guard, environment.
4. Record: OBSERVATION without guard (silent wrong output or misleading KeyError) vs. OBSERVATION with guard (exception type, message, line — quoting the handoff). Then run the valid fixture and record that the guard stays silent (no false trip).

Success criterion. A three-row firing record plus the contract’s code location (file:line, consumer entry) and the exact failure message on the violating repro. A guard that never fired on the pinned repro is not completing the lab — untested tripwires are decorations.

Companion tool: Invariant Contract Checklist

What it accepts: the convicted handoff (Chapters 5–7 output), the one-sentence contract statement, the guard’s code location and mechanism choice with its one-line justification, and the three-row firing record. What it performs: it checks that every convicted handoff has exactly one owning guard, that the guard sits at consumer entry (or documents why elsewhere), that the mechanism fires under the deployment flags (-O audit included), and that the valid-fixture no-trip run is recorded. What it can establish: whether the finding from Chapters 5–7 is now pinned by a loud, local, tested tripwire — and whether that tripwire fired on the pinned repro without tripping on good data. What it cannot establish: output correctness beyond the guarded predicate (a schema guard does not certify arithmetic), producer-side root cause, or coverage of unlisted handoffs — each contract guards its own boundary and no other. How its output changes your next action: an unguarded convicted handoff routes to writing the guard now; a guard that fired routes to committing it with its firing record as the regression test; a guard that tripped on valid data routes to tightening the predicate, not deleting the guard.

Paper form, sufficient for this chapter:

Handoff: producer ___ → consumer ___ (entry line: ___)
Contract (one sentence): ___
Mechanism: assert / raise / schema   Why this one: ___
Fires under -O? Y/N   Cost note: ___
Run 1 (absent × bad input): ___   Run 2 (present × bad input): ___
Run 3 (present × good input): ___  FALSE TRIPS: ___

Where a software implementation does not yet exist in the reader’s stack, this checklist is the tool. The contract discipline precedes any automation.

Reusable procedure: convert every diagnosis into a tripwire

1. Name the handoff from the Chapters 5–7 conviction (producer → consumer, entry line).
2. State the contract in one sentence, as a predicate on the crossing data.
3. Place the guard at consumer entry; choose the weakest mechanism that fires in production.
4. Prove it fires on the pinned violating repro; prove it silent on the valid fixture.
5. Commit guard + firing record together — the test that replays the violation is the prevention artifact Chapter 1 promised.

Failure modes

• Ticket-only knowledge. The fix ships, the contract stays in prose. Six months later the same handoff breaks with a different key.
• Output tests masquerading as contracts. assert summarize(fixture) == report pins one fixture; it does not state the handoff rule and fails opaquely on the next shape.
• Guard at the wrong line. Checking at the crash line (88) instead of consumer entry: the next producer bypasses nothing, but the message names the symptom instead of the contract.
• Silent defaults as guards. .get("refund_id", 0) labeled “defensive.” Defense that hides violations is Chapter 4’s silencing pattern with a new name.
• Unfired tripwires. A guard committed without Run 2’s firing proof. Until it has fired on the pinned repro, it is HYPOTHESIS that the guard guards anything.
• Score-as-contract. Treating a similarity score, a model confidence, or suite greenness as the handoff guarantee. Contracts are predicates on data, checked per run — not statistics about runs.

Limits, per contract: one guard pins one predicate at one handoff under the tested flags; no guard proves general correctness, audits historical corruption, or replaces human verification where money, safety, or production traffic is at stake. UNKNOWN where the predicate itself is disputed — then the deliverable is a spec decision, per Chapter 4.

References

• Bertrand Meyer. Applying “Design by Contract”. Computer 25(10), 1992, pp. 40–51. https://doi.org/10.1109/2.161279
• Michael D. Ernst, Jake Cockrell, William G. Griswold, and David Notkin. Dynamically Discovering Likely Program Invariants to Support Program Evolution. IEEE Transactions on Software Engineering 27(2), 2001, pp. 99–123. https://doi.org/10.1109/32.908957
• Gunnar Kudrjavets, Nachiappan Nagappan, and Thomas Ball. Assessing the Relationship Between Software Assertions and Faults: An Empirical Investigation. Proceedings of the 17th International Symposium on Software Reliability Engineering (ISSRE), 2006, pp. 204–212. https://doi.org/10.1109/ISSRE.2006.14
• Gerard J. Holzmann. The Power of Ten – Rules for Developing Safety Critical Code. Computer 39(6), 2006, pp. 95–99. https://doi.org/10.1109/MC.2006.212

Debugging Checklist

• Convicted handoff named (producer → consumer entry line) from Chapters 5–7 work?
• Contract stated as one executable predicate before coding?
• Guard placed at consumer entry with mechanism justified for deployment flags?
• Three-row firing record: absent-silent, present-loud (type/message/line quoted), present-valid-silent?
• Fix addresses the rule; guard + replay test committed together?
• No .get() silence, output-only test, or score/greenness treated as the contract?

What This Chapter Established

• Contracts as prevention: converting Chapters 5–7 handoff convictions into loud, local tripwires at consumer entry, with mechanism choice (assert/raise/schema) justified per runtime. The precondition/postcondition/invariant vocabulary and the producer-vs-consumer attribution rule are Meyer’s Design by Contract.
• Empirical backing: assertion density correlates with lower fault density in commercial code (Kudrjavets et al.); likely contracts can be inferred from prior passing runs (Daikon), with curation needed for noise; heavy assertion use is the norm in safety-critical code (Power of Ten).
• The silent-drift demonstration: four weeks of refund_id-less rows propagating without exception (H2) vs. late miscomputation (H1), separated by a consumer-entry guard that fails on the first bad row — constructed illustration, no measured runs claimed.
• Lab 8 as a proposed three-row firing record the reader executes; the Invariant Contract Checklist (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: output correctness beyond the guarded predicate, historical audit of past reports, or coverage of any handoff besides the guarded one.

Next

Contracts pin every handoff inside the code — but the code is not the whole system. The pipeline above can carry perfect contracts at every function boundary and still fail, because the failure lives underneath the code: a different interpreter, a drifted dependency, a missing env var, a cache from last month. The next chapter descends to the bottom of Chapter 4’s stack, where identical code and identical data still produce different outcomes.