Part II — Debugging Deterministic Software

The crash that names the wrong culprit

A nightly invoice job dies at 02:14 with this:

Traceback (most recent call last):
  File "jobs/run_invoices.py", line 41, in <module>
    main()
  File "jobs/run_invoices.py", line 36, in main
    totals = summarize(orders)
  File "billing/totals.py", line 88, in summarize
    row["refund_id"] = order["refund_id"]
KeyError: 'refund_id'

The on-call engineer’s eyes lock onto line 88 of totals.py. That is where the traceback ends, so that must be where the bug lives. She adds .get("refund_id"), the job goes green, and three weeks later finance finds $41,000 in refunds silently missing from the report. The traceback told the truth. She read it backwards — or rather, she read only the last line and treated it as the diagnosis.

OBSERVATION: order lacks key refund_id at totals.py:88 on this input, this code version, this container. HYPOTHESIS H1 (code-assumption): summarize assumes a key the upstream stage never guaranteed. HYPOTHESIS H2 (data-regression): the upstream export dropped the column this week; code is unchanged and correct on old data. INFERENCE: none yet — the last frame alone cannot separate H1 from H2.

This chapter’s question: when a traceback lands in your lap, what reading order separates the frame that raised from the frame that diverged first?

Why the obvious reading fails

Three habits convert tracebacks into misdiagnoses:

1. Top-frame fixation. The eye starts at the top (run_invoices.py:41), skims, and lands on the bottom frame (totals.py:88) as “the bug.” But the bottom frame is only where Python gave up. The first divergence — the earliest point where observed execution departed from intent — is usually several frames up, or off the traceback entirely (a caller that passed a malformed dict, a loader that dropped a column).
2. Reading the exception type as the cause. KeyError feels like a verdict: “missing key.” But an exception type names a mechanism (dict lookup failed), not a layer. Chapter 4 showed the same KeyError: 'refund_id' arising at three layers: a code assumption (Chapter 5’s H1), a data regression (H2), an environment drift in CSV parsing (Chapter 4’s H3). The type is constant across all three. Only frame-local evidence separates them.
3. Explaining before freezing. The engineer patched line 88 before saving the input row, the code hash, or the full traceback text. By Chapter 3’s rule, every theory after that moment is storytelling: the evidence that could have falsified H1 vs. H2 is gone.

The cost asymmetry matters. A .get() with a silent default converts a loud crash into quiet corruption. Reading the traceback correctly takes four minutes. Auditing three weeks of corrupted invoices takes three weeks.

The method: bottom-up to locate, top-down to convict

A traceback has four anatomical parts. Learn to name them before you interpret them:

• Exception type and value (last line): the mechanism and the operand. KeyError: 'refund_id' = a mapping lookup for that exact key failed. Nothing more.
• Frames (middle): an ordered call stack, oldest call first, most recent call last. Each frame names a file, a line number, a function, and the source line.
• Frame locals (not printed by default): the actual values in each frame at crash time — retrievable with traceback inspection, pdb post_mortem, or faulthandler. The printed source line shows code text; locals show what the code saw.
• Chained context (During handling of the above exception... / __cause__ / __context__): a second, earlier traceback that the current one was raised from. Always read the root chain first — and note which kind of chain it is. The above exception was the direct cause (__cause__, from an explicit raise X from Y) means a developer deliberately wrapped the root error: the earlier traceback is the diagnosis, the later one is packaging. During handling of the above exception, another exception occurred (__context__, implicit) often means the error handler itself broke while dealing with the first error — the later traceback can be a distraction, and the first one is where to start.

The reading discipline has two passes:

Pass 1 — bottom-up to locate the raising frame. Start at the last line (type + value). Move up one frame: what operation raised, and what were its operands’ runtime values? Confirm the operand, not the code text. order["refund_id"] raises only if order at runtime lacks that key — but is order even a dict? A row object with a custom __getitem__? Locals decide.

Pass 2 — top-down to find the first divergence. Starting from the oldest frame, ask at each handoff: did this frame deliver what the next frame’s contract required? The raising frame is where the program noticed. The diverging frame is where the program departed. The fix belongs to the diverging frame.

    flowchart TD
    T["KeyError: refund_id — the last line"]
    T -->|"Pass 1: bottom-up"| RF["raising frame: totals.py:88 indexes the missing key (where the program noticed)"]
    RF -->|"Pass 2: top-down, check each handoff contract"| DF["first-diverging handoff: clean does not guarantee refund_id to summarize (where the program departed)"]
    DF --> FIX["the fix belongs to the diverging handoff, not the raising line"]
  

OPINION (practitioner heuristic, not a theorem): for application code where a handoff contract is unstated, the fix often lands one to two frames above the raising frame — at the caller that built the bad argument, or the loader that shaped the bad row. Verify per case; do not generalize from this heuristic.

The one large-scale measurement to calibrate against comes from Schröter and colleagues, who matched Eclipse bug-fix commits against the stack traces in their reports. About 40% of bugs were fixed in the frame where the exception was raised, and roughly 88% within the top ten frames; more than 47% of the traces contained at least one method that the fix actually touched (Schröter, Bettenburg & Premraj, 2010). Two lessons. First, the raising frame is the single most likely fix site — “always look one frame up” is as wrong as “always fix the last line.” Second, the tail is long: one bug in eight is fixed outside the ten nearest frames, sometimes off the trace entirely. Read every frame; expect the fix near the bottom; do not stop there.

Demonstration: one KeyError, two causes, one probe

Two incidents, identical last lines, different convicted frames. This is the same-symptom/different-cause pair this chapter owns.

Incident A — code assumption. billing/totals.py:88 indexes order["refund_id"]. Frame-local probe at the caller (jobs/run_invoices.py:36, function main → summarize) shows the upstream clean() output schema never contained refund_id for split-shipment children — on every historical input, including last month’s known-good snapshot. The contract between clean and summarize was never written down; summarize assumed it.

Incident B — data regression. Same last line, same files. But the frame-local probe shows clean() output did contain refund_id for the same fixture last week, and the current input file’s header row lost the refunds join column after an upstream export change. Code version identical (git rev-parse matches); input hash differs.

The discriminating probe is one frame-local check plus one swap, run in this order:

# probe.py — run against the pinned repro input, post-mortem
import traceback, pdb

try:
    from jobs.run_invoices import main
    main("fixtures/incident_input.csv")
except KeyError:
    traceback.print_exc()          # Pass 1: confirm raising frame + value
    pdb.post_mortem()              # Pass 2: inspect locals upward
    # In (pdb): up, p order.keys(), p type(order), up, p clean_output_schema

The current-input column is byte-identical across both incidents; only the old-snapshot swap separates them.

OBSERVATION (constructed illustration, not a measured run): on Incident A, old snapshot + current code still raises; on Incident B, old snapshot + current code passes. UPDATED BELIEF: Incident A convicts the Code layer at the clean → summarize handoff; Incident B convicts the Data layer at the export. The traceback text is byte-identical in both — only the probe separates them.

Note what was not used as evidence: no model’s explanation of the traceback, no similarity score between this traceback and past ones, no “confidence” attached to either hypothesis. Those are downstream artifacts. The diagnosis rests on frame locals and the swap outcome.

Research lineage: traces help, reading is the bottleneck, tooling is catching up

Stack traces measurably speed bug fixing. Schröter and colleagues (above) established both that traces contain the fix location most of the time and where within the trace to expect it. Their study is Java and Eclipse, but a Python-specific replication reaches the same place: Rezaalipour and Furia ran four families of fault-localization technique — spectrum-based, mutation-based, predicate-switching, and stack-trace-based — over 135 real faults in 13 open-source Python projects and reported that the Java findings “largely replicate for Python,” with stack-trace-based localization the fastest family by a wide margin (Rezaalipour & Furia, 2024). This is the empirical warrant for treating the traceback as the primary evidence artifact rather than starting from a code read — and for reading it first, before reaching for a heavier technique.

But developers under-read and under-comprehend them. Barik and colleagues ran an eye-tracking study of 56 developers resolving Java defects and found that participants spent about a quarter of their visual fixations on error messages, that time spent reading the message correlated with fixing the bug, and that compiler error messages scored as hard to read as legal text (Barik et al., 2017). The on-call engineer in this chapter’s opening is the documented pattern: the message was read, glancingly, and its structure was not decoded. The two-pass discipline exists to force the decode.

Python’s own tracebacks have improved under the feet of this method. Since Python 3.11, PEP 657 fine-grained error locations put a caret range under the exact failing sub-expression (order["refund_id"], not just the line), which sharpens Pass 1’s “confirm the operand” step; PEP 678 lets libraries attach add_note() context that travels with the exception. These help locate the raising frame faster. Neither performs Pass 2 — the top-down handoff walk that finds the first divergence is still yours to do.

Asking a model to read the traceback for you is a hypothesis, not a shortcut. Leinonen and colleagues had a code model rewrite real introductory-Python error messages into plain-language explanations: the explanations were comprehensible about 88% of the time, but correct only about 57% of the time they were produced — under half of all inputs — and the incorrect explanations were phrased with the same confidence as the correct ones, sometimes introducing fresh misconceptions (Leinonen et al., 2023). That is Chapter 3’s rule with a number attached: a model’s account of what the traceback means is verbal behavior, readable and roughly coin-flip accurate, and it sounds identical whether it is right or wrong. Use it as a lead to check against frame locals, never as the diagnosis. (The authors’ own suggestion — route by error type and code complexity before invoking the model — is the “propose, then verify” split this book builds toward in Part VIII.)

A stack trace can also become a reproduction. A line of research on crash reproduction (STAR, EvoCrash, and successors) generates a test that reproduces a crash from its stack trace alone. The book’s REPRODUCE step is what those tools automate; where they are unavailable, the pinned repro.py is the manual equivalent.

Lab 5: separate two causes with one frame-local probe

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own probe table.

Setup. Take a script with a 2+ frame traceback ending in KeyError (use the refund_id fixture, or inject one: a loader that optionally drops a key plus a consumer that indexes it directly). Pin the repro: input file hash, code hash (git rev-parse --short HEAD), full traceback saved to evidence/tb5.txt.

Task.

1. Write H1 (code-assumption) and H2 (data-regression) with distinct predictions before opening the debugger — one sentence each, naming the frame you expect to convict.
2. Run Pass 1: record exception type, value, and raising frame (file:line:function + source line).
3. Run Pass 2 with pdb post_mortem or a print of order.keys() / type(order) at the raising frame and the schema at the caller frame. Independent variable: input snapshot (current vs. known-good). Controlled variables: code version, container, command line.
4. Record per run: OBSERVATION (keys present/absent, pass/crash) and UPDATED BELIEF (H1/H2 supported/suspended).

Success criterion. A two-row table plus the convicted handoff named as file:line → file:line (not “line 88 is the bug”). A patch without the table is not completing the lab. If Run 2 passes, you have Incident B; if it still crashes, you have Incident A — either way, the probe, not the last line, decided.

Companion tool: Traceback Inspector

What it accepts: the saved traceback text, the pinned repro reference (input hash + code hash), and the frame-local values recorded at the raising frame and one frame up. What it performs: it enforces the two-pass reading — it will not record a convicted frame until both the raising frame (type/value/operands confirmed) and the top-down handoff walk (each caller contract checked) are filled in, and it flags any diagnosis that cites only the last line. What it can establish: which frame first diverged under this reproduction, and whether the evidence pattern matches a code-assumption profile (fails on old data too) or a data-regression profile (passes on old data). What it cannot establish: the layer below the code (environment drift needs Chapter 9’s container probe), intent gaps (no spec line → UNKNOWN, per Chapter 4), or causality beyond this repro — one convicted frame does not prove no second defect exists. How its output changes your next action: a convicted handoff routes you precisely: code-handoff → read that function’s contract (Chapter 8 writes it down); data-handoff → diff the input snapshot against the fixture; raising-frame-only with no handoff anomaly → suspect the raising frame itself and inspect live values there.

Paper form, sufficient for this chapter:

Raising frame: file:line (fn) | type: ___ value: ___
Locals at raising frame: type(order)=___ keys=___
Caller handoff contract: ___  met? Y/N (evidence: ___)
Old-snapshot swap: crash / pass → convicts H1-code / H2-data
CONVICTED HANDOFF: ___ → ___  EVIDENCE: ___

Where a software implementation does not yet exist in the reader’s stack, this checklist is the tool. The reading discipline precedes any automation.

Reusable procedure: read every traceback this way

1. Freeze the traceback text, input hash, and code hash before anything else (Chapter 3).
2. Pass 1 (bottom-up): name type, value, raising frame; confirm operand values, not code text.
3. Pass 2 (top-down): walk callers oldest-first; mark the first handoff whose contract failed.
4. Hypothesize in pairs: at least one code-assumption and one data-regression hypothesis with distinct swap predictions.
5. Swap one variable: same code × old snapshot (or old code × same input); record OBSERVATION and UPDATED BELIEF.
6. Convert: file the Traceback Inspector note and add the handoff assertion (Chapter 8 makes it permanent).

Failure modes

• Last-line diagnosis. Quoting KeyError: 'refund_id' as the root cause. The last line is the mechanism; the handoff walk is the cause.
• Silent-default patch. .get("refund_id") without establishing which layer dropped the key. Loud crashes are evidence; silenced ones are debt.
• Chained-exception blindness. Reading only the final traceback when a __cause__ chain names the true root two tracebacks up. Always scroll to the first Traceback in the output — and distinguish a __cause__ root (explicit raise ... from, the real diagnosis) from a __context__ chain (implicit, often a broken error handler layered over the real error).
• Single-snapshot conviction. Running only the current input and declaring H1 or H2. Without the old-snapshot swap, both hypotheses predict the identical crash — the experiment has no discriminating power.
• Explanation-as-trace. Pasting the traceback into a model and quoting its summary as the diagnosis. Model text is HYPOTHESIS until frame locals and a swap confirm it — and in the one measured study of exactly this task, the model’s explanations were readable ~88% of the time, correct closer to half, and wrong with the same confident tone (Leinonen et al.).

Limits, per contract: this chapter convicts a frame under one reproduction; it does not prove universal causality, does not certify the environment layer, and does not replace human verification where money, safety, or production traffic is at stake. UNKNOWN where evidence is absent.

References

• Adrian Schröter, Nicolas Bettenburg, and Rahul Premraj. Do Stack Traces Help Developers Fix Bugs? Proceedings of the 7th IEEE Working Conference on Mining Software Repositories (MSR), 2010, pp. 118–121. https://doi.org/10.1109/MSR.2010.5463280
• Mohammad Rezaalipour and Carlo A. Furia. An Empirical Study of Fault Localization in Python Programs. Empirical Software Engineering 29(4), 2024. https://doi.org/10.1007/s10664-024-10475-3
• Titus Barik, Justin Smith, Kevin Lubick, Elisabeth Holmes, Jing Feng, Emerson Murphy-Hill, and Chris Parnin. Do Developers Read Compiler Error Messages? Proceedings of the 39th International Conference on Software Engineering (ICSE), 2017, pp. 575–585. https://doi.org/10.1109/ICSE.2017.59
• Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. Using Large Language Models to Enhance Programming Error Messages. Proceedings of the 54th ACM Technical Symposium on Computer Science Education (SIGCSE), 2023, pp. 563–569. https://doi.org/10.1145/3545945.3569770
• Pablo Galindo Salgado. PEP 657 – Include Fine-Grained Error Locations in Tracebacks. Python Enhancement Proposal, 2021. https://peps.python.org/pep-0657/
• Zac Hatfield-Dodds and Irit Katriel. PEP 678 – Enriching Exceptions with Notes. Python Enhancement Proposal, 2022. https://peps.python.org/pep-0678/
• Ning Chen and Sunghun Kim. STAR: Stack Trace Based Automatic Crash Reproduction via Symbolic Execution. IEEE Transactions on Software Engineering 41(2), 2015, pp. 198–220. https://doi.org/10.1109/TSE.2014.2363469
• Mozhan Soltani, Annibale Panichella, and Arie van Deursen. A Guided Genetic Algorithm for Automated Crash Reproduction (EvoCrash). Proceedings of the 39th International Conference on Software Engineering (ICSE), 2017, pp. 209–220. https://doi.org/10.1109/ICSE.2017.27

Debugging Checklist

• Traceback text, input hash, and code hash frozen before any edit?
• Pass 1 recorded: type, value, raising frame with operand runtime values?
• Pass 2 recorded: top-down handoff walk with first divergence marked?
• H1/H2 (code-assumption vs. data-regression) with distinct swap predictions written before probing?
• One-variable swap run; OBSERVATION and UPDATED BELIEF recorded?
• Convicted handoff named as file:line → file:line, not as the last line alone?
• No model explanation, score, or single-run outcome treated as diagnosis?

What This Chapter Established

• Traceback anatomy (type/value, frames, locals, chained context) and the two-pass reading: bottom-up to locate the raising frame, top-down to convict the first-diverging handoff.
• Empirical calibration: stack traces speed bug fixing and usually contain the fix location (Schröter et al. — ~40% at the raising frame, ~88% within ten frames, ~12% further out), and a Python-specific multi-family study finds the pattern holds and stack-trace reading the fastest localization family (Rezaalipour & Furia); developers read error messages but under-decode them (Barik et al.); a model asked to explain the error is readable ~88% of the time but correct closer to half, with equal confidence when wrong (Leinonen et al.); Python 3.11+ (PEP 657 / 678) sharpens Pass 1 but not Pass 2.
• The same-symptom/different-cause demonstration: identical KeyError last lines from a code assumption (Incident A) vs. a data regression (Incident B), separated by one frame-local probe plus one snapshot swap — constructed illustration, no measured runs claimed.
• Lab 5 as a proposed probe table the reader executes; the Traceback Inspector contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: universal causality from one swap, environment-layer claims, or any diagnosis resting on model-generated text.

Next

A traceback convicts a handoff only when frame-local values confirm what the code text merely suggests — and the printed traceback never shows those values. The next chapter leaves the static text behind and inspects live state instead of guessing it: stopping the program at the convicted handoff, predicting values before looking, and letting runtime observations overrule code reading.