Part III — Debugging Interactive and Numerical AI

The variable that exists nowhere

Chapter 10 closed with a green, in-order Run All — gutter monotonic, cells top-to-bottom. The engineer reopens the same notebook a week later, runs the first six cells interactively, and types threshold in cell 7:

# cell 7 (fresh session, only cells 1-6 executed)
print(threshold)
# 0.9   <- where did this come from?

A search for threshold across every visible cell returns nothing. No definition, no import, no magic. Yet the kernel answers 0.9. Restart & Run All from the top fails at cell 7 with NameError: name 'threshold' is not defined — while the warm kernel insists the variable is real.

OBSERVATION: threshold == 0.9 in the warm kernel; zero definitions in visible cell text; clean Run All raises NameError at the same cell. HYPOTHESIS H1 (deleted-cell residue): threshold was defined by a cell that was later deleted or overwritten — the kernel remembers what the document forgot. HYPOTHESIS H2 (out-of-order / stale-import residue): threshold was defined by a live cell executed under older text, or by a re-imported module whose current source no longer exports it — visible but stale. INFERENCE: none yet — %history, execution_count, and cell text must be cross-checked before either hypothesis earns belief. The kernel’s answer is a MEASUREMENT of state, not an explanation of provenance.

This chapter’s question: when the kernel knows something no visible cell teaches it, what ordered probe names the ghost’s source?

Why the obvious explanation fails

The obvious move — “search the notebook for the definition” — fails because the definition is not in the notebook. It is in the notebook’s past. Three mechanisms plant state no text search can find:

1. Deleted cells. Deleting a cell deletes text, not bindings. threshold = 0.9 ran as [8], the cell was deleted during cleanup, and the binding survived. Chapter 10’s gutter cannot flag what is no longer displayed — the execution sequence has a hole shaped exactly like the missing cell.
2. Overwritten definitions. Cell 4 once read threshold = 0.9; it now reads threshold = load_config()["cutoff"] but was never re-executed after the edit. The kernel holds the old value under the new text. Outputs look current; bindings are fossils with fresh labels.
3. Stale imports and aliased mutation. from utils import threshold ran before utils.py was edited to remove it (module object cached in sys.modules); or df was mutated in place by a cell whose visible text now shows a copy (df_clean = df.copy() added later, never re-run). Re-importing without importlib.reload re-binds nothing — the old module object persists.

OPINION: a long-lived notebook kernel is a crime scene where the perpetrator tidied up. The mess is gone; the fingerprints are in dir().

The mental model: visible cells are the recipe; kernel namespace is the kitchen. Every execution adds ingredients; nothing except an explicit del, a restart, or an overwrite removes them. Debugging hidden state means diffing the kitchen against the recipe and treating every unexplained resident as a suspect.

The method: the fresh-vs-dirty kernel diff

Ordered probe — residue source before content blame:

1. Inventory the kitchen. In the dirty kernel, dump the namespace: %who, dir(), and for suspects threshold — type(), repr(), and id(). Record as MEASUREMENT.
2. Inventory the recipe. Search all visible cell text (and git log -p on the .ipynb) for the suspect name. A definition in history but not in current text supports H1; a definition in current text with mismatched value supports H2.
3. Run the discriminating intervention: fresh-kernel execution to the same cell. Restart, run cells 1→7 top-to-bottom exactly once, and compare. Prediction if H1 (deleted residue): fresh kernel raises NameError — the ghost was history-only. Prediction if H2 (stale redefinition/import): fresh kernel defines the name but with a different value or type (the current text’s version), proving the dirty value was stale. Same value in both kernels would exonerate hidden state entirely — the definition was visible all along and the search was faulty.
4. Convict with provenance, not assertion. The winning evidence is a git diff showing the deleted/edited cell, or a %history line showing the orphaned execution — a pointer to the originating run, not a story about it.

    flowchart TD
    D["dump the dirty kernel: value, type, id() per suspect"] --> T{"name in visible cell text?"}
    T -->|"no"| H1c["H1 candidate: history-only residue"]
    T -->|"yes, value/type mismatches the text"| H2c["H2 candidate: overwritten or stale import"]
    H1c --> FR["restart; run the visible prefix 1..N exactly once"]
    H2c --> FR
    FR --> R{"suspect in the fresh kernel?"}
    R -->|"absent (NameError)"| H1["H1 confirmed: restore the definition in visible text"]
    R -->|"present, differs"| H2["H2 confirmed: fix re-execution / reload policy"]
    R -->|"present, identical"| EX["hidden state exonerated: the text search was faulty"]
    H1 --> P["name the provenance line: %history exec number or git commit"]
    H2 --> P
  

# hidden-state diff probe (run in dirty kernel, save output, then restart and re-run to same cell)
import json
suspects = ["threshold", "df", "region_map"]  # names under investigation
dirty = {name: repr(eval(name))[:200] for name in suspects if name in dir()}
print("DIRTY:", json.dumps(dirty, indent=1))
# After restart + run-to-same-cell, compare against FRESH dict built identically.
# Prediction H1: key present in DIRTY, absent in FRESH (NameError).
# Prediction H2: key present in both, values/types differ.

OBSERVATION (constructed illustration, not a measured run): dirty kernel reports threshold == 0.9 (float); git log -p shows cell 8 threshold = 0.9 deleted in the cleanup commit; fresh kernel raises NameError at cell 7. UPDATED BELIEF: H1 supported — deleted-cell residue. H2 suspended (no live definition exists to go stale). INFERENCE: the fix restores the definition as a visible cell (or deletes the dependence), verified by fresh-kernel Run All — never by del threshold patching on the warm kernel, which hides the recipe gap.

Research lineage: the ghost is a design flaw, not a user error

Every mechanism in this chapter — deleted-cell residue, overwritten-but-not-rerun definitions, stale imports, in-place mutation — is a consequence of one design choice: the notebook uses an imperative REPL where each execution mutates a shared namespace and nothing links the namespace back to the visible code. Two research directions attack that root cause rather than the symptoms.

Make dependencies explicit. Koop and Patel’s dataflow notebooks give each cell a persistent identifier and rewrite in-cell references so the dependency graph between cells is encoded, tracked, and reproducible (Koop & Patel, 2017).

Make execution reactive. Observable (JavaScript), Pluto.jl (Julia), and marimo (Python) run the notebook as a dataflow graph: editing a cell re-runs its dependents automatically, and deleting a cell scrubs its variables from memory. Pluto states the resulting guarantee directly — at any instant, program state is completely described by the code you can see. marimo enforces the same by static analysis, at the cost of forbidding cross-cell mutation and duplicate global names. In those environments this chapter’s bug cannot occur; there is no history for the kernel to remember that the document has forgotten. The tradeoff is real — reactive notebooks give up the freewheeling out-of-order exploration Jupyter allows — but the class of ghost this chapter hunts is designed out rather than detected.

Detect staleness inside Jupyter. Where switching notebook systems is not an option, nbsafety (Macke et al., Chapter 10) brings the same lineage tracking to the standard kernel, flagging exactly the stale and orphaned bindings the fresh-vs-dirty diff hunts by hand.

The practical implication: if you run the fresh-vs-dirty diff more than occasionally on the same project, the finding is not “be more careful” — it is “this workload wants a reactive notebook or a linted kernel.” Discipline is the fix for the incident; tooling is the fix for the class.

Lab 11: fresh-kernel vs. dirty-kernel diff

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own namespace diff.

Setup. Build the ghost deliberately: in a scratch notebook define ghost_var = 42 in a cell, execute it, then delete the cell (or overwrite its text to ghost_var = int(open("config.txt").read()) without re-running). Confirm ghost_var still answers in the warm kernel. You now hold a known-origin ghost — the lab is the detection procedure, not the haunting.

Task.

1. Write H1 (history-only residue: deleted/overwritten cell) and H2 (stale live definition or cached import) with distinct fresh-kernel predictions before restarting.
2. Dump the dirty namespace for the suspect name(s): value, type, id(). Independent variable: kernel history (dirty vs. fresh); controlled variables: file revision, execution prefix (cells 1→N identical), environment.
3. Restart and execute exactly the visible prefix to the same cell, once. Record OBSERVATION (value present/absent/different + verbatim error if any) and UPDATED BELIEF. Consult %history / git log -p to name the originating execution.
4. Repeat the fresh run a second time to exclude one-off ordering slips; divergent repeats are UNKNOWN, not convictions.

Success criterion. A named provenance line (deleted-cell diff or stale-execution number) plus a visible-text fix verified by a green fresh-kernel Run All. A warm-kernel del plus a passing single cell is explicitly not completion — it re-hides the gap.

Companion tool: Hidden State Inspector

What it accepts: the dirty-kernel namespace dump, the current cell texts, execution history (%history / execution_count log), and the git history of the .ipynb. What it performs: it diffs namespace residents against visible definitions, flags names with no live definition (orphans, H1 suspects), names with value/type drift vs. fresh execution (stale, H2 suspects), and cached modules whose source changed since import; it refuses a clean bill while any resident is unexplained. What it can establish: which names are unexplained by visible text and which class of residue (orphan vs. stale) each belongs to — under the examined revision and prefix only. What it cannot establish: data correctness, execution-order health (Chapter 10’s viewer), cross-machine reproducibility (Chapter 12), or intent — an explained namespace can still compute the wrong thing. It never treats namespace agreement or import success as diagnosis. How its output changes your next action: orphan rows route to restoring-or-removing the definition in visible text; stale rows route to re-execution discipline or importlib.reload + import policy; a fully explained namespace with a still-failing run routes to Chapters 12–13 with hidden state exonerated in writing.

Paper form, sufficient for this chapter:

Suspect name: ___   Dirty value/type: ___ / ___
Visible definition? Y/N (cell ___)   History definition? Y/N (exec ___ / commit ___)
Fresh-kernel result: present-absent-different (circle)   value/type: ___ / ___
Cached-module suspect: ___ (source mtime vs import time: ___)
CONVICTION: H1 orphan / H2 stale (circle; provenance line: ___)

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. The namespace-vs-text discipline precedes any automation.

Reusable procedure: every mystery binding gets this

1. Dump before theorizing — value, type, and id() in the dirty kernel.
2. Search text, then history — current cells first, git log -p and %history second.
3. Fresh-kernel prefix run as the discriminating experiment; predictions pre-written.
4. Name the provenance line — execution number or commit that planted the residue.
5. Fix in visible text (restore, remove dependence, or reload policy) and gate with fresh Run All.

Failure modes

• Text-search exoneration. “No definition found, so it must be a kernel bug.” The definition is in history, not in text — absence in current text is the finding.
• Warm-kernel patching. del ghost_var then moving on. The recipe still omits the ingredient; the next warm session re-haunts differently.
• Reload superstition. Re-running import utils and assuming fresh code. Without importlib.reload (or kernel restart), the cached module object persists — re-import is a no-op disguised as a fix.
• In-place mutation blindness. Assuming df is pristine because the cleaning cell “looks functional.” Mutation history lives in execution order, not in current text — re-run the prefix or distrust the frame.
• Single fresh-run conviction. One NameError blamed on cosmic rays or one match treated as proof. Fresh runs are cheap; run twice, record both.
• Agreement-as-provenance. “The value looks right, so the state is fine.” Correct-looking residue is still residue — provenance, not plausibility, is the standard.

Limits, per contract: one namespace diff convicts one residue class under one file revision and prefix; it does not certify ordering (Chapter 10), environment parity (Chapter 9), or future reproducibility. UNKNOWN where history is unrecoverable.

References

• David Koop and Jay Patel. Dataflow Notebooks: Encoding and Tracking Dependencies of Cells. 9th USENIX Workshop on the Theory and Practice of Provenance (TaPP), 2017. https://www.usenix.org/conference/tapp17/workshop-program/presentation/koop
• Stephen Macke, Hongpu Gong, Doris Jung-Lin Lee, Andrew Head, Doris Xin, and Aditya Parameswaran. Fine-Grained Lineage for Safer Notebook Interactions. Proceedings of the VLDB Endowment 14(6), 2021, pp. 1093–1101. https://doi.org/10.14778/3447689.3447712
• Souti Chattopadhyay, Ishita Prasad, Austin Z. Henley, Anita Sarma, and Titus Barik. What’s Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020. https://doi.org/10.1145/3313831.3376729
• Fons van der Plas et al. Pluto.jl: Simple Reactive Notebooks for Julia. 2020–. https://github.com/fonsp/Pluto.jl · Akshay Agrawal et al. marimo: A Reactive Python Notebook. 2023–. https://github.com/marimo-team/marimo

Debugging Checklist

• Dirty-namespace dump (value/type/id()) recorded before theorizing?
• Current-text search + history/git log search both completed?
• H1/H2 with distinct fresh-kernel predictions written before restart?
• Two fresh-prefix runs recorded with verbatim outcomes?
• Provenance line named (exec number / commit / import time)?
• Fix applied in visible text and verified by fresh Run All, not warm-kernel patch?
• No plausible-looking value accepted without provenance?

What This Chapter Established

• Hidden state as kitchen-vs-recipe divergence: deleted cells, overwritten-but-not-rerun definitions, cached imports, and in-place mutations plant bindings no text search finds.
• The fresh-vs-dirty kernel diff (dump → text+history search → predicted fresh-prefix runs ×2 → provenance conviction), demonstrated on the threshold == 0.9 ghost — constructed illustration, no measured runs claimed.
• Lab 11 as a proposed namespace-diff record the reader executes; the Hidden State Inspector contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: execution-order health, cross-machine reproducibility, or computational correctness under an explained namespace.
• The class fix is architectural: dataflow (Koop & Patel) and reactive (Pluto, Observable, marimo) notebooks eliminate these ghosts by construction; nbsafety (Macke et al.) detects them inside Jupyter. Discipline fixes the incident; tooling fixes the class.
• Forward link: order is clean (Ch 10), state is explained (this chapter) — and the notebook still fails on a colleague’s machine. Explained state that only reproduces here is not reproducibility. That discipline is next.

Next

The kernel is clean, the namespace is explained, Run All passes — on this machine, today. Tomorrow it fails on a colleague’s laptop with different package versions, no pinned kernel, and an unseeded shuffle. Nothing in Chapters 10–11 prevents that, because neither chapter pins anything. The next chapter converts today’s green run into tomorrow’s guarantee: reproducible notebooks.