Part III — Debugging Interactive and Numerical AI

What changes in Part III

Parts I and II debugged a program you could re-run: same code, same data, same container, same outcome — or a substrate diff that explained the difference. The object under inspection was small and local — a value at a line, a frame’s state, a handoff between two functions.

Part III keeps the discipline and grows the object. Nothing about the loop changes — reproduce first, find the earliest divergence, move one variable, predict the result before you look, convert the fix to a contract. What changes is what the loop points at, and how hard it is to hold still. The object moves outward, one chapter at a time:

binding → kernel namespace → dataset → tensor pipeline → training system → evaluation instrument

Each of these accumulates hidden state the code that names it does not show, and each behaves less deterministically than the last — so the reproduction bar rises with it: from “run it again” to “restart the kernel,” to “rebuild from the lockfile,” to “three cold runs on two machines,” to “repeat across seeds and report the spread.” By the end of Part III the thing being debugged is no longer a value; it is a system, observed statistically. The method that carried Part II carries all of it unchanged.

Locked env, pinned data, and a notebook that lies anyway

Chapter 9 ended with parity: same code hash, same data hash, same locked container, same outcome. The engineer in this chapter has all of that — pip freeze pinned, git status clean, fixtures hashed. The notebook on screen passes. Every cell shows output, no red tracebacks. Then a colleague clicks Restart & Run All and the same notebook dies on cell 14:

# interactive session (passes)          # Restart & Run All (fails)
[12] df = load("sales_v3.parquet")     [1] df = load("sales_v3.parquet")
[15] df = clean(df, dropna=True)       [2] ...
...                                    ...
[14] report = summarize(df)  # OK      [14] report = summarize(df)
                                       NameError: name 'region_map' is not defined

OBSERVATION: identical file + identical locked environment → passes interactively, raises NameError on Restart & Run All. HYPOTHESIS H1 (stale state): the interactive kernel holds a region_map defined by an earlier, since-edited cell execution that no longer exists in display order. HYPOTHESIS H2 (order dependence): cells were executed in an order that differs from top-to-bottom display, and Run All exposes the true dependency. INFERENCE: none yet — the traceback names the missing variable, not which history produced it. Both hypotheses predict the same NameError under Run All; only an execution-history probe separates them.

This chapter’s question: when the screen shows a working program and Run All shows a broken one, which record do you trust, and what ordered probe convicts display order vs. execution order?

Why “it works on my screen” fails

The obvious explanation — “cell 14 has a bug” — fails because cell 14 is innocent. Three habits make this defect survive review:

1. Reading top-to-bottom. The notebook displays cells 1→14, so the engineer reviews them as a script. But the kernel executed them 1, 2, 12, 15, 14, 13 — the execution_count numbers in the gutter say so. The program that ran is the execution sequence, not the display sequence. Chapter 4’s first-divergence rule applies to the wrong artifact if you read the screen.
2. Re-running cells as patches. A fix is tested by re-running one cell, which appends a new execution ([16]) without invalidating the stale bindings from [12] and [15]. Each re-run layers more history the display never shows. The notebook accumulates a past that no reader can see by scrolling.
3. Trusting outputs as state. Cell outputs are fossils — they record what a cell printed when it last ran, possibly under bindings that no longer exist. A green output under cell 14 proves cell 14 once passed, not that it passes now under current kernel state.

OPINION: notebooks are the only mainstream programming surface where the visible program routinely differs from the executed program. Treat every notebook as suspect until its execution history is inspected.

This is measured, not folklore. Pimentel and colleagues collected 1.4 million Jupyter notebooks from GitHub and tried to run them: only 24% executed without error, and only about 4% produced the same results they originally displayed. Among the notebooks whose intended order was unambiguous, 36% had been executed out of order at least once (Pimentel et al., 2019). A later study raised the stakes and got the same picture: Samuel and Mietchen took ~27,000 notebooks from repositories linked to biomedical papers in PubMed Central — code that a peer reviewer nominally vouched for — installed dependencies where declared, and re-ran the 10,388 that could be attempted; 1,203 finished without error and only 879 (about 8.5% of those attempted) reproduced the results the notebook itself reported (Samuel & Mietchen, 2024). The displayed-vs-executed gap this chapter debugs is the single largest reason a saved notebook does not reproduce.

The method: the display-vs-execution probe

The mental model is simple: a notebook is two programs — the displayed document and the executed sequence — and the kernel only ever ran the second one. Debugging it means reconstructing the second program from execution_count metadata, then testing whether the failure is stale residue (H1) or structural ordering (H2).

    flowchart TD
    G["read the gutter: execution_count, top to bottom"] --> H["freeze the .ipynb, note the hash"]
    H --> R1["Restart and Run All — clean kernel, no history"]
    R1 --> D{"where is the first failing cell?"}
    D -->|"same cell as the interactive suspicion"| H1["H1: stale residue — a deleted/edited-away cell defined it; fix = restore it in visible text"]
    D -->|"an earlier cell"| H2["H2: order dependence — top-to-bottom order violates a real dependency; fix = reorder"]
    H1 --> R2["Restart and Run All again — a clean green run is the only admissible reproduction"]
    H2 --> R2
  

Ordered probe, cheapest exoneration first:

1. Read the gutter, not the screen. Scan execution_count values top-to-bottom. Gaps, repeats, or non-monotonic numbers ([12], [15], [14]) are a MEASUREMENT of out-of-order execution — not a style complaint. Record the true execution sequence on paper.
2. Freeze the display program. Save the .ipynb, note the file hash. This is the artifact Run All will execute top-to-bottom.
3. Run the discriminating intervention: one clean-kernel Run All. Kernel → Restart & Run All, fresh process, no history. Prediction if H1 (stale state): Run All fails (the missing region_map was residue from an edited-away cell); the interactive pass was the artifact. Prediction if H2 (order dependence): Run All also fails but at a different first-divergence cell — the top-to-bottom order itself violates a dependency (e.g., cell 9 uses a frame cleaned in cell 12). Either way Run All is the control; the location of its first failure separates the hypotheses.
4. Localize the first divergence. Under Run All, the earliest failing cell is the conviction point. Everything after it is downstream symptom — do not fix cell 14 if cell 9 is the first red cell under clean execution.

# display-order dependency audit (run in a scratch cell, then delete it)
# MEASUREMENT, not diagnosis: reconstructs the executed program
counts = [(cell_num, nb["cells"][cell_num]["execution_count"]) for cell_num in range(len(nb["cells"]))]
out_of_order = [c for c in counts if c[1] is None or c != sorted(counts, key=lambda x: x[1] or 0)[counts.index(c)]]
print("TRUE EXECUTION SEQUENCE:", sorted(counts, key=lambda x: (x[1] is None, x[1])))

OBSERVATION (constructed illustration, not a measured run): gutter reads [1..11, 12, 15, 14, 13, 16] top-to-bottom; region_map was defined in an earlier version of cell 12 whose current text no longer defines it. Clean Run All fails at cell 12’s successor, not cell 14. UPDATED BELIEF: H1 supported — interactive pass relied on residue from an overwritten cell execution. The fix is a reordering plus re-execution discipline, not a cell-14 logic change. INFERENCE: any notebook fix verified only by single-cell re-run is unverified. Only a clean Run All counts as reproduction (Chapter 4’s rule, notebook edition).

Research lineage: the executed program can be recovered, and tooled

The gap is quantified and its causes are known. Beyond Pimentel’s reproduction rates, Chattopadhyay and colleagues surveyed and interviewed notebook users and found “managing hidden state” and “the mess of nonlinear execution” among the top recurring pain points, not edge cases (Chattopadhyay et al., 2020).

Staleness can be tracked automatically. Macke and colleagues built nbsafety, a drop-in Jupyter kernel that traces fine-grained lineage — which cell defined each variable, and when — and uses liveness analysis to flag cells that are unsafe to run because they depend on stale state, plus cells that would resolve the staleness. Across 666 real notebook sessions it flagged 117 with potential safety errors, and the cells it marked as staleness-resolving were about seven times more likely than chance to be the ones users actually re-ran next (Macke et al., 2021). This is the Execution Order Viewer of this chapter and the hidden-state inspector of the next, built and evaluated.

The minimal reproducing notebook can be extracted. Head and colleagues’ code gathering tools slice a messy notebook down to the minimal ordered set of cells that actually produce a chosen result (Head et al., 2019). That is Chapter 1’s minimization applied to notebook history: once the first-divergence cell is known, gather the slice that feeds it and discard the rest.

The reordering fix can be computed, not guessed. Wang, Kuo, Li, and Zeller’s Osiris parses a notebook’s cells into an abstract syntax tree, does data-flow analysis to recover which cell defines each name and which cells consume it, and searches for an execution order in which every use follows its definition. In their sample of 936 published notebooks that were executable in principle, 73% would not reproduce under straightforward top-to-bottom execution — hypothesis H2 is the common case, not the exotic one — and Osiris reconstructed a valid order for about 82% of the executable set, roughly three times the prior state of the art (Wang et al., 2020). “Reorder-by-intuition” (a failure mode below) is what this replaces: the dependency graph, not the drag-and-drop, decides the order.

Lab 10: display-order vs. execution-order probe

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own Run-All record.

Setup. Take any notebook that passes interactively (or build one: define threshold = 0.9 in cell 8, delete that line, re-run only cell 10 which uses threshold — it still passes on residue). You need the gutter counts and a kernel you are willing to restart.

Task.

1. Write H1 (stale residue: interactive pass depends on overwritten history) and H2 (order dependence: top-to-bottom order itself is broken) with distinct clean-Run-All predictions before restarting. Both predict Run-All failure; they differ on where the first failure lands and whether reordering cells fixes it.
2. Record the gutter sequence (independent variable: execution history — dirty interactive vs. clean Run All; controlled variables: file hash, environment, input data).
3. Restart & Run All exactly once. Record OBSERVATION per the table (first failing cell verbatim + error line) and UPDATED BELIEF. If the Run-All failure cell differs from the interactive suspicion, H2 is live — try one top-to-bottom reorder and Run All again as the follow-up single-variable run.
4. Because kernel timing and async outputs can vary, repeat the clean Run All twice; disagreement across repeats is UNKNOWN, not a conviction — record both.

Success criterion. A written gutter sequence, two clean Run-All records with the first-divergence cell named, and the one-line reorder-or-restore fix verified by a final green Run All. A single-cell re-run green is explicitly not completion.

Companion tool: Execution Order Viewer

What it accepts: the notebook file (cells + execution_count metadata + cell outputs) and the named repro command (Restart & Run All). What it performs: it renders the true execution sequence alongside display order, flags gaps/repeats/non-monotonic counts, marks outputs whose producing execution no longer matches current cell text, and records the first-divergence cell of the latest clean Run All. What it can establish: which program actually ran (the ordered execution list) and where display order and execution order diverge — under the examined file revision only. What it cannot establish: hidden bindings from deleted cells (that is Chapter 11’s inspector), data correctness, environment parity (Chapter 9), or intent truth — a green Run All proves order-consistency, not correctness. It never substitutes agreement or output color for diagnosis. How its output changes your next action: a flagged divergence routes to the reorder-or-restore fix plus a Run-All gate in CI; a clean sequence with a still-failing Run All routes to Chapters 11–12 (hidden state, reproducibility) with order exonerated in writing.

Paper form, sufficient for this chapter:

File hash: ___   Env lock ref: ___
Gutter sequence (display cell → exec count): ___
DIVERGENCE(S): gaps ___ / repeats ___ / non-monotonic ___
Stale-output suspects (output newer than cell text?): ___
Clean Run All #1 first failure: cell ___ err ___
Clean Run All #2 first failure: cell ___ err ___
CONVICTION: H1 stale-residue / H2 order-dependence (circle; evidence line: ___)

A software implementation of much of this already exists: nbsafety (Macke et al., above) replaces the kernel and flags stale and staleness-resolving cells live. Where it is not in the reader’s stack, this record is the tool. The ordering discipline precedes any automation.

Reusable procedure: every notebook gets this before trust

1. Read the gutter — copy the execution sequence before reading any code.
2. Distrust outputs — treat each output as a fossil until a clean run refreshes it.
3. Clean-kernel Run All as the only admissible reproduction; single-cell re-runs are edits, not tests.
4. First divergence under Run All is the defect location; downstream cells are symptoms.
5. Pin the fix — reordered cells committed, or restored definition re-added, gated by a Run-All check (Chapter 12’s checklist entry).

Failure modes

• Gutter blindness. Reviewing cell text without glancing at execution_count. The numbers are the program; the text is the rumor.
• Single-cell verification. “Fixed it — cell 14 passes now” after re-running only cell 14 on a warm kernel. That run inherited the same residue the bug came from.
• Output-as-proof. Screenshots of green cells in a review thread. Outputs persist after the state that produced them is gone.
• Reorder-by-intuition. Dragging cells until it “looks right” without a clean Run All between moves — multiple variables per intervention, Chapter 4’s verdict: inconclusive.
• Single-run exoneration. One green Run All declared as permanent health. Kernel versions, data files, and seeds drift; the gate must run every time (Chapter 12).
• Fixing the symptom cell. Editing cell 14’s summarize() when the first clean-run failure is upstream. Downstream-symptom treatment is never diagnosis.

Limits, per contract: one execution-order record convicts one ordering defect under one file revision; it does not certify hidden-state absence (Chapter 11), data correctness (Chapter 13), or future reproducibility. UNKNOWN where any Run-All row is unmeasured.

References

• João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks. Proceedings of the 16th International Conference on Mining Software Repositories (MSR), 2019, pp. 507–517. https://doi.org/10.1109/MSR.2019.00077
• Souti Chattopadhyay, Ishita Prasad, Austin Z. Henley, Anita Sarma, and Titus Barik. What’s Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020. https://doi.org/10.1145/3313831.3376729
• Stephen Macke, Hongpu Gong, Doris Jung-Lin Lee, Andrew Head, Doris Xin, and Aditya Parameswaran. Fine-Grained Lineage for Safer Notebook Interactions. Proceedings of the VLDB Endowment 14(6), 2021, pp. 1093–1101. https://doi.org/10.14778/3447689.3447712
• Andrew Head, Fred Hohman, Titus Barik, Steven M. Drucker, and Robert DeLine. Managing Messes in Computational Notebooks. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019. https://doi.org/10.1145/3290605.3300500
• Jiawei Wang, Tzu-Yang Kuo, Li Li, and Andreas Zeller. Assessing and Restoring Reproducibility of Jupyter Notebooks. Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2020, pp. 138–149. https://doi.org/10.1145/3324884.3416585
• Sheeba Samuel and Daniel Mietchen. Computational Reproducibility of Jupyter Notebooks from Biomedical Publications. GigaScience 13, 2024, giad113. https://doi.org/10.1093/gigascience/giad113

Debugging Checklist

• Gutter execution_count sequence recorded before any code reading?
• File hash noted; environment parity (Ch 9) confirmed or flagged?
• H1/H2 with distinct first-failure predictions written before restart?
• Two clean Restart-&-Run-All runs recorded with first-divergence cell?
• Fix is reorder-or-restore, verified by final green Run All (not single-cell re-run)?
• No output screenshot or single-run outcome treated as proof?
• Divergence record kept as the regression artifact?

What This Chapter Established

• The displayed notebook vs. the executed program: execution_count order is the program the kernel ran; display order is documentation until a clean Run All promotes it. Empirically the gap is large and durable: ~24% of GitHub notebooks run without error, ~4% reproduce their own results, ~36% ran out of order (Pimentel et al.); ~8.5% of biomedical-publication notebooks reproduce their reported results (Samuel & Mietchen); ~73% of executable published notebooks do not reproduce top-to-bottom (Wang et al.).
• The problem is tooled: nbsafety (Macke et al.) tracks staleness live; code-gathering (Head et al.) slices to the minimal reproducing cell set — Chapter 1 minimization on notebook history; Osiris (Wang et al.) computes a valid execution order from data-flow analysis for ~82% of executable notebooks — the reorder fix, automated.
• The display-vs-execution probe (gutter read → hash file → clean Run All ×2 → first-divergence localization), demonstrated on the passes-interactively/fails-on-Run-All region_map case — constructed illustration, no measured runs claimed.
• Lab 10 as a proposed Run-All record the reader executes; the Execution Order Viewer contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: absence of hidden state (deleted-cell residue is Chapter 11), cross-machine reproducibility (Chapter 12), or data/model correctness under a green Run All.
• Forward link: even a green, in-order Run All can pass on bindings no visible cell defines — residue the gutter cannot show. That invisible remainder is next.

Next

Restart & Run All now passes, cells run top-to-bottom, the gutter is monotonic — and the notebook still cannot be trusted. Somewhere in its history a cell was deleted, a variable redefined out of order, an import shadowed and never re-run: state the kernel holds that no visible cell explains. The next chapter hunts what the execution sequence cannot show — hidden notebook state.