Part II — Debugging Deterministic Software

The suite is green and the refunds are wrong

Chapter 6 taught you to inspect live state on the failing input. This chapter asks why the failing input was never run.

The paginate helper ships with 47 passing tests:

# billing/pages.py
def paginate(items, page_size):
    """Split items into pages of page_size. Returns list of pages."""
    pages = []
    for i in range(0, len(items), page_size):
        pages.append(items[i:i + page_size])
    return pages

Every test uses comfortable middles: 100 items with page size 10, 25 items with page size 5. Production runs it on the refund queue — and one night the queue holds exactly 0 items, then a single-item queue, then a queue whose length equals the page size exactly. The shown paginate itself returns [] correctly for the empty input (its range(0, 0, 10) never executes); the phantom page appears in the production path, where a chunk_count-driven sibling (convicted below) allocates one page too many at every exact multiple — including zero — and downstream code then indexes a page that should not exist. The single-item run works. All 47 tests stay green through all of it.

OBSERVATION: the chunk_count-driven production path allocates 1 page for n=0 (spec expects 0) and an extra trailing page whenever len(items) is an exact multiple of the page size. The plain paginate listing above is correct for []; it is the seam arithmetic in the sibling that diverges. HYPOTHESIS H1 (empty-input defect): the loop/range construction mishandles the zero-length case only. HYPOTHESIS H2 (fencepost defect): the page-count arithmetic is off by one at every exact multiple, of which empty is one instance. INFERENCE: none yet — the two passing-test suites look identical; only boundary probes separate H1 from H2.

This chapter’s question: when all normal tests pass, what small fixed set of boundary probes tells a single-point defect apart from a broken rule?

Why “normal” tests fail to fail

Three habits keep boundary bugs invisible:

1. Middle-only sampling. Test authors pick representative values — 100 items, tier “premium,” a mid-range date. Representative values exercise the steady state, where off-by-one errors hide. The defect lives at 0, 1, N-1, N, N+1 — the values nobody finds representative.
2. One-shape fixtures. Every test reuses the same 10-row CSV, the same 3-order fixture from Chapter 4. A single shape can only confirm a single shape. Boundary clustering — empty, single, exact-multiple, just-over, huge — requires deliberately deformed fixtures, and deforming a fixture feels like breaking something that works.
3. Passing as proving. “47 tests pass” is read as “pagination is correct.” But a suite of middles proves middles. Chapter 3’s rule applies: the claim “handles all lengths” predicts outcomes at lengths 0, 1, N-1, N, N+1. Until those rows exist in the table, the claim is HYPOTHESIS, and the green badge is a downstream symptom mistaken for a diagnosis.

The mental model: bugs cluster at boundaries because boundaries are where one rule hands off to another — empty vs. non-empty, partial vs. full page, range stop vs. slice end. Chapter 4’s handoff idea recurs at miniature scale: every < vs. <=, every range(0, n, k) vs. manual page count, is a contract between two regimes. Probe the handoff, not the regime interior.

One more idea does the real work of this chapter: failure shape. A single failing input is an observation. A pattern of failing inputs is evidence about the generating rule. “Production fails at n=0” is one data point compatible with many defects. “Fails at 0, 10, 20, 30 and nowhere else” is a shape, and the shape names the mechanism — an off-by-one that fires exactly when the remainder is zero. The boundary table exists to turn the first kind of fact into the second.

    flowchart TD
    F["one failing input (e.g. n=0)"] --> S["run the fixed stencil: 0, 1, N-1, N, N+1, large, plus a recurrence probe 2N"]
    S --> SH[read the failure shape across the cells]
    SH --> ONE{"one divergent cell?"}
    ONE -->|yes| G["single-point defect: targeted guard + regression test on that cell"]
    ONE -->|no| M["rule defect: rewrite the arithmetic/comparison, re-run the whole stencil"]
  

The method: the boundary table

For any integer-shaped input (length, index, count, page), run this fixed set before any other test design:

• Empty (0 / [] / ""): does zero produce zero outputs, or one phantom?
• Single (1 element): does the minimal non-empty case take the general path or a degenerate one?
• Just below the seam (N-1 where N is the page/window/limit size).
• Exactly at the seam (N).
• Just above the seam (N+1).
• Large (a stress multiple, e.g. 10_000): does the pattern hold or does a second regime (chunking, batching) intrude?

Write the table before running, with a prediction per cell derived from the spec — not from the code. Then run and mark divergences. The convicted hypothesis is the one whose predicted divergence pattern matches the observed marks.

When the spec is silent about the seam — no line says what chunk_count(0, 10) should be — a metamorphic relation can still act as an oracle: a property that must hold between two runs even when neither run’s exact output is specified. For a pager: chunk_count(k * size, size) == k for all positive k; chunk_count(n, size) <= chunk_count(n + 1, size); chunk_count(n, size) - chunk_count(n - 1, size) in (0, 1). A divergence from one of these convicts the arithmetic without any absolute oracle.

But the relation is itself a claim about intent, and it needs its own warrant. Monotonicity and unit-step increments follow from what “a page count” means and are safe to assert. chunk_count(k * size, size) == k assumes an exact multiple allocates no empty trailing page — reasonable for a pager, but if that is the very question in dispute, the relation cannot settle it. The rule: where the exact output is unspecified but an independently justified relation is known, use the relation as the oracle; where neither the output nor a relation can be justified, the seam stays UNKNOWN and routes to a specification decision (Chapter 4), not a guessed assertion.

Demonstration on a realistic fencepost variant — the kind that passes middles because middles never land on the seam:

# billing/chunks.py — the convicted sibling (constructed illustration)
def chunk_count(n, size):
    """How many pages for n items at page size `size`?"""
    return (n // size) + 1          # defect: exact multiples get a phantom page

OBSERVATION (constructed illustration, not a measured run): divergences at n=0, 10, 20 — the exact multiples including zero. UPDATED BELIEF: H2 (fencepost) supported; H1 (empty-only) suspended — H1 predicts n=10 and n=20 pass, and they do not. INFERENCE: empty is not a special case here; it is the N=0 instance of the general off-by-one. The fix is the ceiling-division rule (n + size - 1) // size, not an if not items special-case patch that would leave n=10 and n=20 broken.

Why the rule fails exactly at multiples. (n // size) + 1 carries a hidden assumption: that integer division always threw away a non-zero remainder, so one more page is always needed for the leftovers. For n = 11 (11 // 10 == 1, remainder 1) that holds — 1 + 1 = 2 is right. For n = 9 it holds trivially — 0 + 1 = 1. The assumption is true across the whole interior of every regime. It is false at exactly the points where the remainder is zero: n = 10, 20, 30. The ceiling form (n + size - 1) // size adds the “+1 page” before dividing, so it only rounds up when there is a real remainder to round.

What the FAIL PASS PASS FAIL PASS FAIL shape supports: the defect is not empty-only (10 and 20 fail too); failures align with exact multiples, which points at division/remainder arithmetic; a zero-only patch is falsified as sufficient. What it does not support: correctness after the fix on negative n, size = 0, huge inputs, or any other seam — the table convicts one rule under one code version, nothing more.

Note the fix discipline: H1’s patch (if n == 0: return 0) silences the reported instance and preserves the defect at every other multiple — Chapter 4’s silencing anti-pattern at boundary scale. H2’s fix corrects the rule. The table is what tells you which patch you are writing. (The ceiling form already returns 0 for n = 0 when size > 0; state the zero case as a contract for readers, but it needs no special arithmetic branch.)

The same table shape applies to off-by-one slices, < vs. <= comparisons, empty-string vs. None handling, and single-element collections. The seam value changes; the six-row discipline does not.

Research lineage: why the six-cell table earns its place

Boundary faults are single-parameter faults, and those are the largest class. Kuhn, Wallace, and Gallo examined failure reports across several domains and formulated the interaction rule: most failures are triggered by one or two parameter values interacting, with single-parameter conditions the single biggest category (Kuhn, Wallace & Gallo, 2004). That is the quantitative reason this chapter’s table varies one input at a time across its range: the highest-yield defects are exactly the ones a one-variable sweep exposes.

The table is a hand-written property test. Claessen and Hughes’s QuickCheck asks the programmer to state a property — chunk_count(k * size, size) == k — and then generates and runs cases against it, and on failure shrinks the counterexample to a minimal one (Claessen & Hughes, 2000). The six-cell table is the manual version; a property-based test is the automated version, and its shrinking step is Chapter 1’s minimization applied to test inputs. Where a property-based framework is available (Hypothesis for Python), write the metamorphic relations above as properties and let it find the seam:

from hypothesis import given, strategies as st

def chunk_count(n, size):
    return (n // size) + 1                      # the defect

@given(k=st.integers(1, 1000), size=st.integers(1, 1000))
def test_exact_multiples(k, size):
    assert chunk_count(k * size, size) == k     # fails, shrinks toward k=1, size=1

The framework searches the space the six cells sampled by hand, and on a failure it shrinks the counterexample toward the smallest one — here k=1, size=1, i.e. chunk_count(1, 1) == 2 — which is the seam stated in its plainest form.

A green suite is a weak signal. Inozemtseva and Holmes found only a low-to-moderate correlation between code coverage and a suite’s ability to detect faults once suite size is controlled for, and that stronger coverage criteria did not improve the picture (Inozemtseva & Holmes, 2014). A follow-up by Zhang and Mesbah found that the number of assertions per test correlated with effectiveness far better than coverage did (Zhang & Mesbah, 2015). “47 tests pass” tells you little; “47 tests with seam-cell assertions pass” tells you something.

Metamorphic relations are the oracle substitute this book leans on for learned systems. The technique has its own survey (Chen et al., 2018), and it was carried into machine learning early: Xie, Ho, Murphy, Kaiser, Xu, and Chen enumerated metamorphic relations that a classifier must satisfy — permuting the order of training examples, scaling or shifting all feature values, adding an uninformative attribute, duplicating instances — none of which requires knowing the correct label for any single prediction, and a violation is a genuine defect (Xie et al., 2011). Their case study found real faults in Weka’s kNN and naive-Bayes implementations this way. The chunk_count relations in this chapter are the same move on deterministic arithmetic; Parts IV–VII apply it where there is never a per-output oracle — sampling distributions, retrieval faithfulness, agent trajectories.

Lab 7: the boundary table that separates H1 from H2

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own boundary table.

Setup. Take any function with an integer-shaped input and a suspected boundary defect (use chunk_count above, paginate, or your own: a slicer, a pager, a retry-with-limit helper). Pin the repro: code hash, spec line for the seam behavior (or UNKNOWN if the spec is silent — then the lab’s first deliverable is naming the gap, per Chapter 4’s Intent layer).

Task.

1. Name the seam: the constant where regimes hand off (page size, limit, window, index bound).
2. Write H1 (single-point defect: only one boundary value diverges) and H2 (rule defect: a pattern of values diverges) with distinct per-cell predictions before running. Minimum cells: empty/zero, single, seam-1, seam, seam+1, large.
3. Run the six cells. Independent variable: input magnitude/shape (one cell per run). Controlled variables: code version, environment, all other arguments.
4. Record OBSERVATION per cell (actual return value verbatim) and UPDATED BELIEF per hypothesis after the full row.

Success criterion. A completed six-cell table plus the named pattern (single-point vs. rule defect) and the fix that addresses the pattern, verified by re-running the full table after the patch. A single failing input with a patch is not completing the lab — the table is what proves the patch addressed the rule rather than the instance.

Companion tool: Boundary Inspector

What it accepts: the function signature, the named seam constant, the six-cell table with pre-run predictions, and the post-run observations. What it performs: it enforces table completeness — no verdict until all six cells have predictions written before their observations — and it matches the divergence pattern against single-point vs. rule-defect profiles, flagging H1-style single-cell patches when the pattern spans multiple cells. What it can establish: which boundary values diverge under this code version, and whether the evidence profile is a single-point defect or a rule (fencepost) defect. What it cannot establish: correctness on non-integer-shaped inputs (strings, encodings, time zones need their own seam analysis), intent truth when the spec is silent (marks UNKNOWN and routes to a spec decision), or absence of further defects — a clean table on one seam does not certify adjacent seams. How its output changes your next action: a single-point verdict routes to a targeted guard with a regression test on that cell; a rule-defect verdict routes to rewriting the arithmetic/comparison and re-running the full table — never to a one-cell special case.

Paper form, sufficient for this chapter:

Function: ___  Seam constant: ___ (= ___ )
Cell | Input | Predicted (spec) | Observed | Diverges? Y/N
empty| ___   | ___              | ___      | ___
single|___   | ___              | ___      | ___
seam-1|___   | ___              | ___      | ___
seam |___    | ___              | ___      | ___
seam+1|___   | ___              | ___      | ___
large|___    | ___              | ___      | ___
PATTERN: single-point / rule-defect   EVIDENCE: ___

Where a software implementation does not yet exist in the reader’s stack, this table is the tool. The boundary discipline precedes any automation.

Reusable procedure: boundary-first testing

1. Name the seam from the code or spec (page size, limit, bound, capacity).
2. Write the six cells with spec-derived predictions before running.
3. Run one cell per experiment, recording verbatim outputs.
4. Match the pattern: single divergent cell → targeted guard; multiple/seam-spanning divergences → rewrite the rule.
5. Re-run the full table after the fix; file the Boundary Inspector sheet as the regression artifact.

Failure modes

• Middle-only suites. Dozens of passing tests, zero seam cells. Coverage of interiors proves nothing about handoffs.
• Instance patching. Fixing n=0 with a special case while n=10 and n=20 still diverge. The table exists to make this visible before the patch ships.
• Code-derived predictions. Predicting what the code will return (by mentally executing it) instead of what the spec requires. That converts the table into a tautology: the code always agrees with itself.
• Single-cell conviction. Running only the reported input (n=0) and declaring H1. Without seam±1 and the next multiple, H1 and H2 predict identically — no discriminating power.
• Correlation-as-diagnosis. “Failures correlate with large inputs, so it must be scale.” Boundary tables separate magnitude effects (large-only divergence) from seam effects (multiples diverge at any size) — correlation alone cannot.

Limits, per contract: one table convicts one seam under one code version; it does not certify other seams, does not resolve silent spec gaps (UNKNOWN), and does not replace human verification where money, safety, or production traffic is at stake.

References

• D. Richard Kuhn, Dolores R. Wallace, and Albert M. Gallo. Software Fault Interactions and Implications for Software Testing. IEEE Transactions on Software Engineering 30(6), 2004, pp. 418–421. https://doi.org/10.1109/TSE.2004.24
• Koen Claessen and John Hughes. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. Proceedings of the 5th ACM SIGPLAN International Conference on Functional Programming (ICFP), 2000, pp. 268–279. https://doi.org/10.1145/351240.351266
• Laura Inozemtseva and Reid Holmes. Coverage Is Not Strongly Correlated with Test Suite Effectiveness. Proceedings of the 36th International Conference on Software Engineering (ICSE), 2014, pp. 435–445. https://doi.org/10.1145/2568225.2568271
• Yucheng Zhang and Ali Mesbah. Assertions Are Strongly Correlated with Test Suite Effectiveness. Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE), 2015, pp. 214–224. https://doi.org/10.1145/2786805.2786858
• Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. Metamorphic Testing: A Review of Challenges and Opportunities. ACM Computing Surveys 51(1), 2018, article 4. https://doi.org/10.1145/3143561
• Xiaoyuan Xie, Joshua W. K. Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. Testing and Validating Machine Learning Classifiers by Metamorphic Testing. Journal of Systems and Software 84(4), 2011, pp. 544–558. https://doi.org/10.1016/j.jss.2010.11.920

Debugging Checklist

• Seam constant named explicitly (value + what regimes it separates)?
• Six cells (empty/single/seam-1/seam/seam+1/large) with spec-derived predictions written before running?
• One cell per run; code version and environment held constant?
• Verbatim outputs recorded per cell; divergences marked against spec, not code?
• Metamorphic relations written where the spec is silent (instead of stopping at UNKNOWN)?
• Single-point vs. rule-defect pattern named with supporting cells cited?
• Fix addresses the pattern; full table re-run green after patch?
• Seam-cell assertions added to the suite (not just coverage of the seam)?
• No green-suite, correlation, or single-cell outcome treated as certification?

What This Chapter Established

• Boundary clustering: deterministic defects concentrate at regime handoffs (empty/single/seam/large), and middle-only suites cannot detect them. Empirically, single-parameter conditions are the largest class of fault triggers (Kuhn et al.’s interaction rule) — which is why a one-variable sweep is high-yield.
• Failure shape: a single failing input is an observation; a repeating pattern of failing inputs (FAIL PASS PASS FAIL PASS FAIL → the exact multiples) is evidence about the generating rule. The buggy (n // size) + 1 assumes integer division always discarded a remainder — true in every regime interior, false exactly where the remainder is zero.
• The six-cell boundary table with pre-run spec predictions, demonstrated on the chunk_count fencepost: identical empty-case failure from H1 (empty-only) vs. H2 (rule defect), separated by the seam and multiple cells — constructed illustration, no measured runs claimed.
• The table is a manual property test; property-based testing (Claessen & Hughes) automates the search and shrinks failures to minimal cases; metamorphic relations cover spec-silent seams (Chen et al.) and extend to learned systems with no per-output oracle (Xie et al. found real faults in Weka classifiers via permutation/scaling relations) — the bridge to Parts IV–VII.
• A green suite is a weak signal: coverage is not strongly correlated with fault detection (Inozemtseva & Holmes); assertion count is (Zhang & Mesbah).
• Lab 7 as a proposed boundary table the reader executes; the Boundary Inspector contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: correctness beyond the probed seam, spec truth where the spec is silent, or any certification resting on suite greenness or input correlation.

Next

Boundary tables catch the defect once — but nothing in the codebase prevents its return. The next regression, the next refactor, reopens the same seam, and the table sits in a chapter file instead of in the code. The next chapter converts findings like these into artifacts that fire on their own: assertions, invariants, and contracts at the handoffs where Chapters 5–7 kept finding breaks.