Part III — Debugging Interactive and Numerical AI

Green here, red there, same notebook

Chapters 10 and 11 closed with a notebook that runs clean: order verified, namespace explained, Run All green. The engineer sends it to a colleague. Same file, same Run All — different number:

# author laptop (passes)                  # colleague laptop (fails)
Cell 18: val_accuracy = 0.91              Cell 18: val_accuracy = 0.84
Run All: green                            Run All: green — but different answer
# and on the author's own machine, rerun: 0.88

No traceback. No ghost variable. Three executions, three answers, zero errors. Worse: re-running on the same machine drifts too.

OBSERVATION: identical .ipynb hash → 0.91 / 0.84 / 0.88 across two machines and one rerun; all runs green. HYPOTHESIS H1 (unpinned substrate): kernel spec, package versions, or data file differ across runs — the program is deterministic but its inputs are not the same. HYPOTHESIS H2 (unpinned nondeterminism): unseeded randomness (shuffle, init, dropout, GPU ordering) makes the same program a different draw each run. HYPOTHESIS H3 (unlocked data/time): the notebook reads data/latest.csv, calls datetime.now(), or streams an API — the world moved between runs. INFERENCE: none yet — green runs with divergent numbers convict nothing by themselves. Only pinning one variable class per run separates H1/H2/H3.

This chapter’s question: when the same notebook gives different answers without failing, what ordered discipline converts “runs” into “reproduces”?

Why “it ran” is not “it reproduces”

The obvious explanation — “floating point / GPUs are just nondeterministic, accept the noise” — fails because most notebook irreproducibility is administrative, not numerical. Four unpinned inputs do the damage before randomness matters:

1. Unpinned kernel and packages. The notebook metadata names python3 — which resolves to 3.10 with sklearn 1.2 on one machine and 3.11 with sklearn 1.4 on the other. Chapter 9’s freeze diff applies verbatim: the environment is an input, and >= specifiers make every install a new experiment.
2. Unseeded randomness. train_test_split(...) without random_state, numpy/random/torch seeds unset, PYTHONHASHSEED drifting (set-order iteration changes), GPU kernels nondeterministic by default. Each rerun is a fresh draw presented as a replication.
3. Unlocked data. pd.read_csv("data/latest.csv") resolves to different bytes per machine and per week. No hash, no version, no chance of reproduction — the input file is a moving target wearing a fixed name.
4. Time and network. datetime.now() in a feature, requests.get(...) in a loader, random.sample over a changing directory listing. Wall-clock and network reads import the outside world into a supposedly closed computation.

OPINION: reproducibility is not a virtue of careful people; it is a checklist of pinned things. Unpinned items are not minor gaps — each one is a license for the result to differ.

The mental model: a reproducible notebook is a pure function of (code hash, lockfile, seed set, data hash, pinned time). Change any argument, change the result — legitimately. The discipline is making every argument explicit, then demonstrating green Run All from a cold start on a second substrate with all five pinned.

The method: the reproducibility checklist run

Ordered probe — substrate before stochasticity before world, cheapest pin first, one variable class per run:

1. Pin the kernel and environment. Record kernelspec name, pip freeze hash, and re-run from a fresh env built from the lockfile. Prediction if H1: the cross-machine gap collapses once both sides build from the same lock (remaining drift is H2/H3).
2. Pin every seed. random.seed, numpy, framework seeds, PYTHONHASHSEED=0, deterministic flags where affordable; log all seed values in cell 1. Rerun twice on the same machine. Prediction if H2: rerun variance collapses (repeated trials agree); persistent variance implicates GPU nondeterminism or H3. One caveat for the model chapters ahead: pinned seeds buy isolation, not bitwise replay — batch-non-invariant serving can still vary outputs at temperature 0 (Ch21). Collapsed variance means the seeds were the mechanism, not that replay is now exact.
3. Lock data and time. Hash every input file (sha256sum), vendor the bytes or pin the versioned URI, freeze TZ=UTC and inject a fixed timestamp instead of now(). Prediction if H3: the rerun and cross-machine numbers converge once data hashes match.
4. Cold-start Run All on two substrates. Fresh kernel, cleared outputs, Run All on machine A and machine B (or container vs. host). Reproducibility is demonstrated agreement, not asserted hygiene — record both numbers.

    flowchart TD
    B["baseline: two cold Run Alls, record both numbers, declare tolerance"] --> S1["pin substrate: kernelspec + lockfile, rebuild a fresh env"]
    S1 --> C1{"cross-machine gap collapses?"}
    C1 -->|yes| H1["H1: unpinned substrate"]
    C1 -->|"narrows, drift remains"| S2["pin seeds: random / numpy / framework / PYTHONHASHSEED=0"]
    H1 --> S2
    S2 --> C2{"rerun variance collapses?"}
    C2 -->|yes| H2["H2: unseeded nondeterminism"]
    C2 -->|"persists"| S3["lock data + time: sha256 inputs, vendor bytes, TZ=UTC, injected timestamp"]
    H2 --> S3
    S3 --> C3{"numbers converge across runs and machines?"}
    C3 -->|yes| H3["H3: unlocked data / time"]
    C3 -->|no| GPU["residual GPU-kernel nondeterminism: document it, bound it with repeated trials"]
    H3 --> CF["confirm with 3 cold runs, report the spread, enforce in CI"]
    GPU --> CF
  

# cell 1: the reproducibility header (every reproducible notebook opens with this)
import random, os
SEED = 42
random.seed(SEED)
os.environ["PYTHONHASHSEED"] = "0"
import numpy as np
np.random.seed(SEED)
# torch.manual_seed(SEED)  # uncomment per stack; log versions below
print("seed =", SEED, "| data_sha =", open("evidence/data.sha256").read().strip())
# MEASUREMENT: seed set + data hash printed into the run record itself

OBSERVATION (constructed illustration, not a measured run): freeze diff showed sklearn 1.2 vs 1.4 (H1 live); after lockfile rebuild the gap narrowed 0.91/0.84 → 0.89/0.88; after seeding + vendored sales_v3.parquet (sha256:9f2c…) two cold Run Alls agreed at 0.885 ± 0.002 over three repeated trials. UPDATED BELIEF: H1 + H2 + H3 all contributed — no single cause. The checklist converted three red rows to green one variable class at a time. INFERENCE: “accept the noise” would have hidden two administrative defects behind one stochastic excuse. Pinning order matters: substrate first, or seed experiments measure the wrong distribution.

Because nondeterminism is intrinsic here, every claim in this chapter requires repeated trials: no convergence statement on fewer than three cold runs; single-run agreement is UNKNOWN, not green.

Two goals, two opposite uses of the seed. Pinning the seed is the right move for debugging: it makes the run a deterministic function you can bisect, so a change in output means a change in code, not a change in draw. It is the wrong move for trusting the number. Bouthillier and colleagues show that a conclusion drawn from a single seed — “model A beats model B” — frequently reverses under a different seed, so a pinned-seed val_accuracy is one sample, not an estimate (Bouthillier, Laurent & Vincent, 2019). Their quantitative follow-up went further: across five deep-learning tasks, data splitting and weight initialization were among the largest variance sources, and — counter-intuitively — randomizing many sources at once (splits, init, data order, augmentation, dropout) estimated true generalization performance better than the “ideal” fix-everything-but-one procedure, at roughly fifty times less compute (Bouthillier et al., 2021). This chapter pins the seed to isolate the administrative defects; Chapters 14–16 deliberately un-pin it — and the lesson from the follow-up is to un-pin broadly, varying every stochastic source, not just the RNG seed, because that is what the number actually is.

Research lineage: the checklist has a track record

Independent reproduction succeeds about two times in three, and the checklist items are what predict it. Raff manually re-implemented 255 machine-learning papers spanning 1984–2017 without looking at any released code, and found roughly 63% independently reproducible. The features that correlated with success were the ones this chapter pins: detailed pseudocode, fully specified hyperparameters, and narrowly scoped empirical claims; sheer mathematical density did not help (Raff, 2019).

The “reproducibility checklist” is a real instrument. The NeurIPS 2019 Reproducibility Program introduced a machine-learning reproducibility checklist and a code-submission policy; the follow-up report describes what changed and what did not (Pineau et al., 2021). This chapter’s Notebook Reproducibility Checklist is a narrowed, notebook-scoped version of that instrument.

The words matter. The literature distinguishes repeatability (same team, same setup, same result), reproducibility (a different team runs the original artifacts and gets the same result), and replicability (a different team, new artifacts, same finding). Chapters 10–12 are chasing reproducibility in that strict sense: a colleague, your artifacts, your number. Replicability — does the finding survive a fresh implementation — is a Part-VI concern.

Lab 12: checklist run, one variable class at a time

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own red-to-green checklist record.

Setup. Take the Chapter 10–11 notebook (or inject irreproducibility: remove one seed, point the loader at an unversioned latest.csv, install two package versions across two envs). You need two runtimes or two cold runs minimum.

Task.

1. Write H1/H2/H3 with distinct predictions before pinning anything: which pin collapses which gap, and by roughly how much (FORECAST with tolerance, e.g., “lockfile rebuild halves the cross-machine gap”).
2. Baseline: two cold Run Alls, record both numbers. Independent variable: pinned class (substrate → seeds → data/time); controlled variables: file hash, Run-All procedure, seed values once set.
3. Apply pins one class per run, recording OBSERVATION after each (numbers verbatim) and UPDATED BELIEF. Never pin two classes between measurements — multi-variable jumps are inconclusive by Chapter 4’s rule.
4. Close with three confirmatory cold runs; report spread (min/max), not a single figure.

Success criterion. A fully green checklist (every row pinned with artifact hashes) plus three cold Run Alls agreeing within a pre-declared tolerance. A single green pair after multi-class pinning is explicitly not completion — the per-class attribution rows are the diagnosis.

Companion tool: Notebook Reproducibility Checklist

What it accepts: the notebook hash, kernelspec + lockfile/freeze artifacts, seed set + PYTHONHASHSEED, input data hashes, clock/network policy, and the cold-Run-All number table (≥3 runs). What it performs: it enforces pin order (substrate → seeds → data/time), refuses a reproducibility verdict while any row is UNKNOWN, checks repeated-trial spread against the declared tolerance, and stamps the green record with all five hashes. What it can establish: that the notebook reproduces within tolerance across the examined substrates and reruns — and which pin closed which gap, per the staged table. What it cannot establish: data correctness (Chapter 13), model/eval validity (Chapters 15–16), or future stability — today’s green rots with the next dependency release unless CI re-runs the checklist. It never treats one green run or headline-version match as reproducibility. How its output changes your next action: a green record routes to CI enforcement (Run All from lockfile on every change, hashes asserted); any red row routes to its pin (lockfile, seed header, vendored data, frozen clock) with the staged table as the regression artifact.

Paper form, sufficient for this chapter:

Notebook hash: ___   Kernelspec: ___   Lockfile/freeze hash: ___
Seeds (random/numpy/torch/PYTHONHASHSEED): ___ / ___ / ___ / ___
Data hashes: ___   Clock (frozen UTC / injected ts): ___   Network reads: none / pinned (circle)
Cold Run Alls (≥3): ___ ___ ___   spread ___ ≤ tol ___? Y/N
RED→GREEN per class: substrate ___ seeds ___ data/time ___ (evidence line each)

Where a software implementation does not yet exist in the reader’s stack, this checklist is the tool. The pinning discipline precedes any automation.

Reusable procedure: every notebook destined for sharing gets this

1. Baseline the gap — two cold Run Alls, numbers recorded, tolerance declared.
2. Pin substrate — kernelspec + lockfile, fresh-env rebuild, re-measure.
3. Pin seeds — full seed set header, deterministic flags, re-measure with repeats.
4. Lock data/time — hash inputs, vendor bytes, freeze clock, stub network, re-measure.
5. Confirm ×3 cold and enforce in CI — green record with all hashes, re-run per change.

Failure modes

• Seed-only superstition. Setting random.seed(42) while packages and data float. Seeds pin the draw, not the distribution — substrate first.
• Latest.csv syndrome. Version-less data paths. A fixed name on moving bytes is the most common H3 and the least examined.
• Headline-version parity. “Both run Python 3.11.” Chapter 9 already convicted this: the freeze file is the environment; headlines are rumors.
• Single-pair green. Two agreeing runs after a three-class pin jump. Agreement without attribution teaches nothing and prevents nothing.
• Output-committed notebooks. Committing executed outputs as proof of health. Outputs are fossils (Chapter 10); the checklist record is the proof.
• Tolerance amnesia. Reporting 0.885 without ± spread or run count. A number without its repeat distribution is a FORECAST wearing MEASUREMENT’s clothes.

Limits, per contract: one checklist record certifies the examined revision across the examined substrates within the declared tolerance; it does not certify correctness, does not freeze the future, and does not replace human verification where money, safety, or production traffic is at stake. UNKNOWN where any row is unmeasured or trials < 3.

References

• Edward Raff. A Step Toward Quantifying Independently Reproducible Machine Learning Research. Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. https://papers.nips.cc/paper/2019/hash/c429429bf1f2af051f2021dc92a8ebea-Abstract.html
• Xavier Bouthillier, César Laurent, and Pascal Vincent. Unreproducible Research is Reproducible. Proceedings of the 36th International Conference on Machine Learning (ICML), 2019, pp. 725–734. https://proceedings.mlr.press/v97/bouthillier19a.html
• Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gaël Varoquaux, and Pascal Vincent. Accounting for Variance in Machine Learning Benchmarks. Proceedings of Machine Learning and Systems (MLSys), 2021. https://arxiv.org/abs/2103.03098
• Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché-Buc, Emily Fox, and Hugo Larochelle. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). Journal of Machine Learning Research 22(164), 2021, pp. 1–20. https://www.jmlr.org/papers/v22/20-303.html
• Hans E. Plesser. Reproducibility vs. Replicability: A Brief History of a Confused Terminology. Frontiers in Neuroinformatics 11:76, 2018. https://doi.org/10.3389/fninf.2017.00076

Debugging Checklist

• Two cold-Run-All baselines recorded with pre-declared tolerance?
• Kernelspec + lockfile pinned, fresh-env rebuild measured (H1 row)?
• Full seed set + PYTHONHASHSEED pinned, repeats measured (H2 row)?
• Data hashed/vendored, clock frozen, network stubbed, re-measured (H3 row)?
• Pins applied one class per run — no multi-class jumps?
• Three confirmatory cold runs with spread ≤ tolerance?
• Green record kept with all five hashes for CI enforcement?

What This Chapter Established

• Reproducibility as a five-argument function (code, lockfile, seeds, data, clock) with ordered pinning: substrate → seeds → data/time, each attributed by its own measurement.
• The checklist run converting the 0.91/0.84/0.88 divergence to attributed green rows — constructed illustration, no measured runs claimed; repeated-trials rule (≥3 cold runs) for all nondeterminism claims.
• Lab 12 as a proposed red-to-green record the reader executes; the Notebook Reproducibility Checklist contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: correctness of data, model, or evaluation under a green record — reproducibility is necessary, not sufficient.
• Empirical grounding: independent reproduction of ML papers succeeds ~63% and is predicted by exactly these pinned items (Raff); the checklist is a real instrument (Pineau et al., NeurIPS 2019); pinning the seed serves debugging, not performance estimation — a single-seed number is one sample (Bouthillier et al. 2019), and a trustworthy estimate randomizes many stochastic sources (splits, init, order, augmentation), which is both more accurate and far cheaper than fixing all but one (Bouthillier et al. 2021).
• Forward link: a notebook can now reproduce faithfully — and faithfully compute on corrupt inputs. Deterministic garbage, delivered identically everywhere. The next defense is upstream of all of this: the data.

Next

The notebook reproduces. Same numbers, every machine, every rerun — and the numbers may still be lies, because reproducibility never checked what the numbers mean. A leaked label, a corrupt split, a silently shifted schema: all reproduce perfectly and all train a model on fiction. Chapters 10–12 debugged the container of computation; the next chapter debugs its contents — the data, before the model.