Part I — Debugging From First Principles

The puzzle: three suspects, one KeyError

Chapter 3 taught you to trust only frozen artifacts. This chapter tells you where to look for them — in what order, and at which layer.

Scenario. A batch job crashes:

KeyError: 'refund_id'
  File "report.py", line 90, in build_row
    row["refund_id"] = order["refund_id"]

Three engineers propose three fixes within minutes:

• Engineer A: “Line 90 assumes a key that clean doesn’t guarantee. Guard with .get().” (Code layer.)
• Engineer B: “The upstream export dropped the column yesterday. Re-fetch the data.” (Data layer.)
• Engineer C: “Works on my machine — the job container pins pandas 1.5, prod runs 2.x, and the CSV reader changed dtype inference.” (Environment layer.)

OBSERVATION: order lacks refund_id at line 90 on this input, this code version, this container. HYPOTHESES: H1 code-assumption, H2 data-regression, H3 environment-drift. INFERENCE: none yet — all three predict the identical traceback.

Same symptom, three layers, three different owners. Without a layer order, the team debates opinions. With one, they run a sequence that eliminates layers cheapest-first. That sequence is the debugging stack.

Why layer order matters

Chapters 1–3 gave you the loop (observe → reproduce → minimize → localize → hypothesize → experiment → verify → prevent), the search rule (first divergence), and the hygiene (evidence before explanation). None of them answers the question this chapter owns: when the first divergence could live at any of several layers, which do you rule out first, and what evidence rules out a whole layer at once?

The naive answer — start where you have expertise — is how code experts “fix” data problems with .get() guards that silence the crash and corrupt the report, and how data experts “fix” environment problems by re-exporting files that were never broken. Each local fix moves the symptom and preserves the defect. Chapter 2’s checkpoint rule generalizes here: rule out layers top-down or bottom-up, but always in stack order, never by jumping to the most familiar layer.

The stack for this book, from symptom to substrate:

┌ 7. Symptom      — what the user reports ("report wrong / job crashed")
├ 6. Behavior     — intended vs. observed values at the boundary (Ch 1)
├ 5. State        — intermediates in execution order (Ch 2)
├ 4. Code         — the program text that computes state
├ 3. Data         — inputs, snapshots, schemas the code consumes
├ 2. Environment  — interpreter, packages, OS, config, secrets
└ 1. Intent       — the spec itself (is "correct" even defined?)

Two readings. Top-down (7→1) starts from the report and descends until evidence contradicts a layer’s contract. Bottom-up (1→2→3…) verifies the substrate first when drift is suspected (“nothing changed in code, so start below it”). The cue picks the direction: if something recently changed — a deploy, a dependency bump, a new data source — enter at that layer; if nothing changed, descend from the symptom. Either direction is legitimate; skipping layers is not. The most expensive mistake in debugging is a confident fix at the wrong layer.

Two orderings are in play and they answer different questions. Stack order decides what you may not skip — you cannot convict Code while Data and Environment sit untested, because all three predict the identical traceback. Probe cost decides the sequence among the layers still in play: a git stash and re-run (Code) costs seconds, a known-good-snapshot re-run (Data) costs minutes, a fresh locked container (Environment) costs longer — so the demonstration below probes Code first even though it is not the top of the stack. The stack forbids the jump; cost orders the walk.

Note the Intent layer at the bottom. Sometimes the stack resolves to “the spec never defined refund_id for split shipments” — no layer is broken because no layer was ever told what correct means. That outcome is a finding, not a failure: it converts a bug ticket into a spec decision, which is Chapter 1’s prevention artifact in another form.

Building the mental model: each layer has a contract and a probe

A layer is debuggable only if it states a contract that evidence can falsify. Vague layers (“the data is bad”) are not layers — they are stories. Concretely:

The power move is the layer-swap probe: hold every layer constant except one, and observe. Same code + known-good data → passes? Then the divergence lives in Data, and all Code hypotheses are suspended. Same data + previous code version → passes? Then it lives in Code. One variable per experiment (Chapter 1’s invariant) applied at layer granularity.

The precondition the layer-swap probe needs: separability. Just as bisection needs monotonicity (Chapter 2), the layer-swap needs layers that can actually be held constant one at a time. Deterministic code has that property; machine-learning systems fight it. Sculley and colleagues named the phenomenon CACE — “Changing Anything Changes Everything”: in an ML system, a change to a hyperparameter, a data-selection rule, or an upstream feature propagates through the model in ways that resist isolation (Sculley et al., 2015). So the probe is crisp in Part II (deterministic software), and by Parts IV–VII you are usually swapping a layer and measuring a shift in a distribution rather than a binary pass/fail. The stack still orders the search; the verdicts just get noisier.

    flowchart TD
    S[Symptom: KeyError refund_id] --> B[Behavior: repro pinned?]
    B --> ST[State: which checkpoint first misses key?]
    ST --> C[Code: does V-1 pass same input?]
    C -->|yes| FIXC[defect in Code layer]
    C -->|no| D[Data: does known-good snapshot pass?]
    D -->|yes| FIXD[defect in Data layer]
    D -->|no| E[Env: does locked container pass?]
    E -->|yes| FIXE[defect in Environment]
    E -->|no| I[Intent: is refund_id even specified?]
  

Demonstration: the refund_id incident, resolved in order

Pinned reproduction first (Behavior): a 3-order fixture, one split shipment, crashes deterministically. Checkpoint (State): clean output schema shows refund_id present for normal orders, absent for split-shipment children — first divergence at the clean → build_row handoff, before line 90 executes. So line 90 is effect; the handoff contract is cause.

Layer probes, cheapest first:

• Code probe. Run fixture against previous code version (same data, same container).
  • Prediction if H1 (code): V-1 passes. Prediction if H2/H3: V-1 also crashes.
  • OBSERVATION (constructed illustration): V-1 passes — clean in V-1 propagated refund_id to children; current V dropped it during a refactor.
  • UPDATED BELIEF: H1 strongly supported. H2/H3 suspended — but not deleted, because a code defect and a data drift can coexist.

The fix therefore belongs at the Code layer (restore propagation + schema assertion), not as a .get() silence, not as a data re-fetch. And the prevention artifact asserts the layer contract: clean output must validate against the schema before build_row ever runs.

H1: code-assumption (V broke propagation). H2: data-regression (column dropped upstream). H3: environment-drift (pandas inference). TEST: V-1 × same-data probe, then same-code × known-good-snapshot probe, then locked-container probe — one layer per experiment. Prediction table recorded before running; each outcome eliminates at most one layer; the surviving layer owns the fix.

Had the Code probe failed (V-1 also crashes), the next probe swaps Data: same code on last week’s known-good snapshot. Pass → data regressed, investigate the export. Fail → descend to Environment: fresh locked container. Pass → drift confirmed (pip freeze diff names the suspect). Fail everywhere → ascend no further; descend to Intent: quote the spec for split-shipment refund_id. If the spec is silent, the deliverable is a spec amendment, and every “fix” proposed before that moment was premature.

Research lineage: the layers are real, and they leak

The stack is a teaching device, but each layer corresponds to a fault class that empirical studies keep rediscovering.

Faults cluster by layer. Humbatova and colleagues built a taxonomy of real deep-learning faults from 1,059 GitHub artifacts and Stack Overflow posts plus practitioner interviews, and the top-level categories — model/architecture, training, tensors and inputs, GPU usage, API — map almost one-to-one onto this chapter’s Code, Data, and Environment layers (Humbatova et al., 2020). A taxonomy is evidence that “which layer?” is a real, recurring first question, not a stylistic one.

The Data layer is the most under-debugged and the most compounding. Sambasivan and colleagues interviewed 53 practitioners and found that 92% had experienced a data cascade — a data problem that stays invisible until it surfaces, badly, far downstream — yet data work is consistently the least glamorous and least instrumented part of the system (Sambasivan et al., 2021). A second taxonomy points the same way from a different corpus: Islam and colleagues studied 2,716 Stack Overflow posts and 500 GitHub fix commits across five deep-learning libraries and found data bugs plus logic bugs the most severe types (over 48%), with the Data Preparation stage the most bug-prone pipeline stage and crash the major effect (Islam et al., 2019). Their cause categories (wrong model parameters, structural inefficiency) cut across this chapter’s layers rather than sitting inside one — the stack is a search order, not a fault ontology — but their API-change bugs (breakage from backward-incompatible releases) are exactly the Environment-drift class this chapter’s H3 names. This is Chapter 2’s symptom-distance point at organizational scale: the crash is at line 90, the cause is three teams upstream, and nobody was watching the boundary.

The stack instantiates per domain. For retrieval systems specifically, Barnett and colleagues catalogued seven recurring failure points across three production RAG deployments — missing content, top-ranked document missed, relevant chunk not in the assembled context, answer not extracted from present context, wrong format, wrong specificity, incomplete answer (Barnett et al., 2024). Those are the “state” and “code” layers of a RAG pipeline enumerated in execution order — exactly the checkpoint chain Part IV will build.

Together: the stack is not arbitrary, its layers entangle as you move toward ML (CACE), and the Data layer deserves probes it rarely gets.

Lab 4: one symptom, three layers, one probe each

Setup. Take the refund_id scenario (or inject an analogous defect into your own 2+ stage script: a dropped key, a changed CSV column, a version-sensitive parse). Record code hash, input hash, and dependency lock.

Task. Write all three layer-hypotheses with distinct predictions before touching anything. Then run the probes in stack order, one layer per experiment:

• Independent variable per run: exactly one swapped layer (code version, or input snapshot, or container).
• Controlled variables: everything else pinned.
• Stop rule: the first layer whose swap flips the outcome owns the investigation; deeper layers wait.
• Record OBSERVATION per probe and UPDATED BELIEF per hypothesis.

Success criterion. A layer-elimination note: “V-1 × same data: still crashes → Code not exonerated / not convicted alone…” through to the convicted layer — plus the contract assertion you added at the convicted boundary. Finding the fix without the note is not completing the lab.

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own elimination table.

Companion tool: Debugging Stack Map

What it accepts: the layer list (behavior → state → code → data → environment → intent), the pinned repro reference, and per-layer contract statements with their ruling-out probes. What it performs: it renders the stack with each layer’s verdict (untested / exonerated / convicted / UNKNOWN) and enforces order — you cannot mark a layer convicted while the layer above it is still untested, and a fix recorded at one layer must link to the probe that exonerated the others. What it can establish: which layer the first divergence belongs to under this reproduction. What it cannot establish: the line-level cause within the layer (that is Chapters 5–9 work), nor Intent-layer truth when no spec exists — it marks that UNKNOWN and routes to a spec decision. How its output changes your next action: a convicted layer confines all subsequent reading, logging, and hypothesis work; exonerated layers are off-limits until re-checkpointing after the fix suggests a second break.

Paper form, sufficient for this chapter:

[ ] Behavior pinned (repro.py + input hash + version)?
[ ] State checkpoints ordered, first break marked?
[ ] Code: V-1 × same-data probe → ______
[ ] Data: same-code × good-snapshot probe → ______
[ ] Environment: locked-container probe → ______
[ ] Intent: spec line quoted or UNKNOWN declared?
CONVICTED LAYER: ______  EVIDENCE: ______

Where a software implementation does not yet exist in the reader’s stack, this map is the tool. The visualization discipline precedes any automation.

Failure modes

• Favorite-layer jumping. The code expert patches .get(), the data engineer re-exports, the platform engineer rebuilds the image — each without ruling out the other layers. The stack exists to make this visible.
• Silencing vs. fixing. A .get() with a default that hides a contract violation converts a loud crash into silent corruption. Loud is better; assert at the handoff instead.
• Single-probe generalization. One exonerated layer on one fixture does not certify the layer universally. Re-verify with the original failing snapshot plus the suite.
• Spec-skipping. Treating an Intent-layer gap as a Code bug. If no spec line defines the expected output, writing code first just encodes someone’s guess.

Limits, per contract: convicting a layer on one reproduction does not prove universal causality; no stack map converts a vendor claim or a model explanation into evidence; human verification remains required before shipping the fix where money, safety, or production traffic is at stake.

References

• D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NeurIPS), 2015. https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems
• Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. Taxonomy of Real Faults in Deep Learning Systems. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE), 2020, pp. 1110–1121. https://doi.org/10.1145/3377811.3380395
• Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M. Aroyo. “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, article 39. https://doi.org/10.1145/3411764.3445518
• Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. A Comprehensive Study on Deep Learning Bug Characteristics. Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2019, pp. 510–520. https://doi.org/10.1145/3338906.3338955
• Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. Seven Failure Points When Engineering a Retrieval Augmented Generation System. Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering (CAIN), 2024, pp. 194–199. https://doi.org/10.1145/3644815.3644945

Debugging Checklist

• Symptom restated as measurable behavior with pinned repro?
• Checkpoints identify the handoff where the break first appears?
• Layer hypotheses (code / data / env / intent) with distinct predictions written?
• Layers actually separable for this system, or is the swap a distribution shift (CACE)?
• Probes run in stack order, one swapped layer each?
• Convicted layer owns all further investigation?
• Handoff contract assertion added at the convicted boundary?
• Data layer given a real probe, not assumed innocent?
• Intent gap declared UNKNOWN rather than coded around?

What This Chapter Established

• The debugging stack (symptom → behavior → state → code → data → environment → intent) with per-layer contracts and cheapest-first ruling-out probes.
• The layer-swap discipline: one swapped layer per experiment, recorded predictions, ordered elimination — and its precondition, separability, which ML systems erode (Sculley et al.’s CACE).
• Empirical backing: fault taxonomies cluster by layer (Humbatova et al.), two independent DL-bug studies put data faults among the most severe and the Data Preparation stage the most bug-prone (Sambasivan et al.’s data cascades, 92% prevalence; Islam et al.’s 2,716-post / 500-commit study), and the stack instantiates per domain (Barnett et al.’s seven RAG failure points) — with the caveat that a cause taxonomy (Islam’s) can cut across the stack’s layers, because the stack is a search order, not a fault ontology.
• The refund_id walkthrough and Lab 4 as constructed/proposed exercises — no measured production results claimed.
• Part I’s closing map: Chapters 1–3 gave loop, search, and hygiene; this chapter gives the terrain.

Next

Part I ends here. Its machinery assumed short, readable, deterministic programs where every layer fits on one screen. Part II, “Debugging Deterministic Software,” applies the machinery where practitioners actually live: reading Python tracebacks (Chapter 5), inspecting live state instead of guessing (Chapter 6), and hunting the boundary conditions and contracts where deterministic bugs cluster.