Part X — The Debugging AI Playbook

The incident that owes a published record

Chapter 58 ended with isolation and queued artifacts: one defect, one repair candidate, deeper questions deferred. Now the incident that defeats the hour: a weekend of duplicate refunds across hundreds of accounts, two services disagreeing on whose handoff dropped scope, a regulator asking what happened, and leadership asking what prevents recurrence. Triage routed it; the hour isolated one edge; neither suffices. This incident owes customers remediation, the team prevention, and the record an account — with named roles producing named artifacts on a visible schedule.

OBSERVATION: duplicate authorizations span many accounts over two days; service A’s logs show schema-valid sends, service B’s logs show scope-less receipts; no shared request ID joins the two; guardrail verdict events are absent on the disputed edge. HYPOTHESIS H1 (contract failure): sends violate the edge schema (A owns the repair). H2 (consumption failure): sends are valid; B’s consumption diverges from protocol (B owns the repair). H3 (coordination/evidence failure): the join is missing so ownership is undecidable from current records — the first repair is instrumentation, not assignment. INFERENCE: none yet — H1/H2/H3 predict different boundary-and-join signatures and separate only under the full procedure with roles and frozen artifacts.

This chapter’s question: what is the complete procedure — roles, artifacts, schedule, and publication — that converts a full incident into certified prevention?

Why “thorough heroics” fails first

The obvious move — the best engineer investigating exhaustively alone — fails because completeness without roles produces evidence nobody trusts. Six defects hide behind solo thoroughness:

1. Role collapse. Investigator, evidence custodian, communicator, and repair owner in one head. Each duty corrupts the others; the timeline becomes self-certifying.
2. Artifact improvisation. Each incident invents its own record shapes. Cross-incident comparison dies; the toolkit (Ch60) has nothing uniform to index.
3. Blame-first routing. Ownership assigned before H1/H2/H3 separate. The assigned team then investigates its own innocence — findings discounted in advance.
4. Silent remediation. Customers remediated without a published account. Trust billed without receipt; the regulator returns.
5. Prevention by promise. “We will add checks” with no clause versions, calibration sets, or CI links. Promises are not guardrails (Ch54’s rule, enforced here at incident scale).
6. Unbounded investigation. No schedule, no definition of done. The incident stays open for a quarter, blocking the conveyor with perfectionism.

OPINION: a full investigation without named roles and a publication date is a private education billed as organizational learning. Staff it, schedule it, publish it.

The mental model: the full investigation as the book’s method at full depth — every chapter’s instrument played once, by its assigned role, into a published packet. KEPT at this budget (nothing skipped except generality claims): Ch52 records assembled across services with joins repaired; Ch53 conveyor run to bidirectional tests; Ch54 clauses registered and calibrated per committing edge; Ch55 joint ledger with the incident’s cost booked; Ch56 live record audited; Ch57–58 records consumed as opening artifacts. SKIPPED honestly: cross-incident generality (“this can never happen again” is forbidden — the packet claims this defect class on these edges under these clauses, nothing more).

The published packet is the book’s single diagnostic-case record (Chapter 45) at its last stage — the same object that was a seven-slot crash dump at capture, a hypothesis record in Chapter 1, a case file in Chapter 24, an evidence ledger in Chapter 34, a trajectory in Chapter 36, a preregistered ledger in Chapter 49 — now an account with named roles, cited records, and a publication date.

This is the structure of a formal accident investigation. The NTSB separates the factual report (what the recorders show, published first, no analysis) from the analysis and the probable cause finding, states contributing factors as a separate list rather than collapsing everything into one cause, and is barred by statute from apportioning blame or liability — its sole purpose is preventing recurrence. Every one of those is a rule this chapter imports: manifest before theory, H1/H2/H3 held plural, blame routed by record signature not advocacy, prevention as the deliverable.

The method: roles, phases, and the published packet

Staff four roles, run five phases on schedule, publish one packet:

1. Roles (day 0). Incident commander (schedule, order, publication), evidence custodian (freezes, hashes, fidelity labels — diagnoses nothing), investigator (hypotheses, interventions, trials), prevention owner (clauses, tests, rollout — named before causation is known, so prevention has an author regardless of blame routing). No role shares a head with another on SEV-1.
2. Phase 1 — Assemble (days 0–1). Custodian freezes all bundles, repairs joins (shared IDs backfilled only where logs permit; the rest marked UNKNOWN-fidelity), and publishes the evidence manifest. Nothing theorizes before the manifest.
3. Phase 2 — Isolate (days 1–3). Investigator runs H1/H2/H3 with pre-written signatures across boundary records (Ch43’s routing), environment bisection, and minimization — ≥3 trials per cell, distributions recorded. Blame routes by record signature, never by terminal position or team advocacy.
4. Phase 3 — Repair and calibrate (days 3–5). Prevention owner registers guardrail clauses per committing edge with calibration sets and thresholds (Ch54), promotes bidirectional regression tests (Ch53), and books the joint cost ledger for each repair (Ch55). Each repair ships single-variable with pre-written joint predictions.
5. Phase 4 — Remediate and publish (days 5–7). Commander oversees customer remediation from the frozen scope list, then publishes the packet: timeline from hashes, H1/H2/H3 verdicts with deciding records cited verbatim, repairs with bidirectional verdicts, clauses with calibration sets, ledger deltas, residual risks named, and the generality boundary stated. INFERRED labeled throughout; UNKNOWNs named, never backfilled.

    flowchart TD
    R["day 0: staff four separated roles — commander, evidence custodian (diagnoses nothing), investigator, prevention owner (named before causation is known)"] --> P1["Phase 1 (d0-1): custodian freezes all bundles, repairs joins, publishes the evidence manifest — nothing theorizes first"]
    P1 --> P2["Phase 2 (d1-3): investigator runs H1/H2/H3 by RECORD SIGNATURE across boundary records + bisection + minimization, >=3 trials per cell"]
    P2 --> RT{"blame routes by signature, never by terminal position or team advocacy"}
    RT -->|"send violates the edge schema"| H1["H1 contract failure — sender-side repair"]
    RT -->|"valid input, deviant consumption"| H2["H2 consumption failure — receiver-side repair"]
    RT -->|"join missing / ordering-sensitive"| H3["H3 coordination + evidence failure — instrumentation ships as its own clause"]
    H1 --> P3["Phase 3 (d3-5): prevention owner registers calibrated guardrail clauses per committing edge, promotes bidirectional tests, books the joint cost ledger — one variable per repair"]
    H2 --> P3
    H3 --> P3
    P3 --> P4["Phase 4 (d5-7): commander oversees remediation from the frozen scope list, then publishes the packet — INFERRED labeled, UNKNOWNs named, generality boundary stated"]
  

FULL INVESTIGATION PACKET (published per SEV-1/selected SEV-2):
incident ___ | roles: commander ___ custodian ___ investigator ___ prevention ___
evidence manifest: bundles ___ (frozen ___ / reconstructed-UNKNOWN ___), joins ok/broken ___
H1/H2/H3 verdicts: ___ / ___ / ___ (deciding records cited verbatim ___)
repairs: ___ (pre ___/___ red, post ___/___ green, x3) | clauses: ___ v___ (cal set ___)
ledger: Q___ L___ S___ deltas ___ | remediated accounts ___ | residual risks ___
GENERALITY BOUNDARY: this class, these edges, these clauses. Nothing universal.
RULE: unlabeled inference anywhere in the packet reopens the investigation.

OBSERVATION (constructed illustration, not a measured run): boundary checks show service A sends missing the scope field against schema v2 — H1-shaped; service B’s consumption follows its protocol given violating input (H2 exonerated here); the missing join delayed routing by two days (H3 as structural finding with its own instrumentation repair). UPDATED BELIEF: H1 supported for this instance (edge-contract failure, A-side repair); H2 exonerated here; H3 supported as process cause (join repair ships as its own clause with its own calibration). Blame routed, evidence repaired, prevention plural — the packet holds all three without merging them.

No team narrative (“service B should have coped”), no executive confidence, no inter-team agreement as verdict, no single confirming run, and no downstream calm (“refunds stopped”) enters the packet as evidence. Records route; prose accounts.

Example: the weekend duplicates, fully worked

The staffed team executes phases without role collapse:

# full procedure: manifest, isolate, repair, publish (roles separated throughout)
manifest = custodian.freeze(services=["A", "B"], window="48h")  # OBSERVATION only
print(manifest.joins)  # MEASUREMENT: A->B join broken, fidelity labeled per bundle
# Investigator: H1/H2/H3 with pre-written boundary signatures, >=3 trials per cell
verdicts = isolate(manifest, hypotheses=[H1_contract, H2_consumption, H3_join],
                   trials=3, cite_verbatim=True)  # blame by signature, never by position
# Prevention owner: one clause + one test per repair, each single-variable, joint ledger
for repair in verdicts.repairs:
    register_clause(repair.edge, calibrate_on=frozen_set, trials=3)  # Ch54 discipline
    promote_test(repair.repro, bidirectional=True, trials=3)  # Ch53 discipline
    book_ledger(repair, quantities=["Q", "L", "S"])  # Ch55 discipline
publish(packet(manifest, verdicts, repairs, ledger, residuals))  # commander; INFERRED labeled

In the constructed case the packet ships on day seven: schema v3 with reject-and-ask on the A→B edge, the join key repaired across services with backfill-fidelity labels, regression tests bidirectional-green, calibration sets hashed into the clauses, remediation scoped from frozen records, and two residual risks named (unlisted-agent edges, traffic-regime threshold drift). The licensed claim covers this incident’s class on these edges — stated on the packet’s final page, where generality goes to be bounded.

Lab 59: packet drill with pre-written routing predictions (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own retrospective packet.

Setup. Take one significant past incident (or a multi-service staged failure). Staff all four roles — solo readers rotate roles explicitly with role-labeled notes, never merged. The procedure (habitual postmortem vs. this chapter’s phases) is the independent variable; incident and records are controlled.

Task.

1. Before assembling, write H1/H2/H3 with distinct predicted boundary-or-join signatures and name the deciding record each requires.
2. Run the phases on a compressed schedule (one day per phase maximum); the custodian’s manifest precedes all theorizing; every cell runs ≥3 trials where records permit.
3. Publish a packet a fresh reader can audit: every verdict traceable to a cited record, every repair bidirectional, every UNKNOWN named.

Success criterion. A published packet with manifest, verbatim-cited verdicts, bidirectional repairs, calibrated clauses, ledger deltas, remediation scope, residuals, and a stated generality boundary. A thorough narrative without these artifacts is explicitly not completion.

Companion tool: Full Incident Investigation Checklist

What it accepts: the role roster, the evidence manifest with fidelity labels, H1/H2/H3 with pre-written signatures, trial logs, boundary and bisection records, repair candidates with joint predictions, calibration sets, ledger inputs, and the draft packet. What it performs: it verifies role separation, manifest-before-theory ordering, single-variable interventions with declared trials, verbatim record citation per verdict, bidirectional test proof per repair, calibrated clauses per committing edge, joint ledger per repair, remediation scoping from frozen records, and generality bounding — reopening the investigation on any unlabeled inference. What it can establish: whether the incident is fully accounted and its prevention certified — for the examined incident, edges, and clauses only. What it cannot establish: cross-incident immunity, threshold permanence across traffic regimes, or completeness over unknown classes. It never treats narratives, confidence, agreement, single runs, or calm as packet evidence. How its output changes your next action: packet-complete routes to publish and schedule threshold review; gaps route to the owning phase with one queued intervention; residuals route to the backlog with named monitors — each tracked by ID.

Paper form, sufficient for this chapter:

Incident ___ | roles staffed separately y/n ___ | manifest ___ bundles (UNKNOWN: ___)
H1/H2/H3: ___ / ___ / ___ (records cited ___)
Repairs: ___ (bidir ___/___) | Clauses: ___ (cal ___) | Ledger Q/L/S: ___/___/___
Remediated ___ | Residuals ___ | Generality: this class, these edges. PUBLISHED ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Staff it, schedule it, publish it.

Research lineage: formal accident investigation and systemic causation

The NTSB template. Formal transportation-accident investigation supplies this chapter’s skeleton. The party system assigns distinct organizations and individuals to distinct working groups (structures, powerplants, operations, human factors) rather than one investigator covering everything — the four-role separation. The factual report is published before any analysis and contains no probable-cause finding — manifest before theory. The final report states a probable cause plus a separately enumerated list of contributing factors — H1/H2/H3 held plural, not merged. And the Board’s rules bar its investigations from being “conducted for the purpose of determining the rights, liabilities, or blame of any person or entity” (49 CFR § 831.4(c)), with a companion statute keeping its reports out of damages suits (49 U.S.C. § 1154(b)): the investigation exists to prevent recurrence, which is exactly why this chapter routes blame by record signature and names a prevention owner before causation is known.

Systemic causation (STAMP/CAST). Leveson’s Engineering a Safer World (MIT Press, 2011) argues that in software-intensive systems, accidents rarely reduce to a single broken component — they result from inadequate enforcement of safety constraints across a control structure. CAST (Causal Analysis based on System Theory) investigates why the constraints and feedback loops were inadequate, not just which part failed. This is the research basis for H3 as a first-class outcome: “the join was missing so ownership was undecidable” is a control-structure defect, and it ships its own clause with its own calibration rather than being folded into the A-side or B-side repair. CAST has been applied to the 737 MAX, Fukushima, and the 2008 financial crisis — all cases where single-cause framing failed.

Blameless analysis, again. The Chapter 56 lineage (Allspaw’s blameless postmortems on Dekker’s Just Culture; Cook on hindsight and root-cause-as-artifact) carries directly into the packet: “no team narrative enters as evidence,” “INFERRED labeled throughout,” and “residuals exist, name them” are the written-artifact form of blamelessness at incident scale.

Bounds: the NTSB model is for physical-transportation accidents with regulated recorders and legal independence; a company’s internal investigation has neither subpoena power nor statutory blame immunity, so the “no blame” property has to be a deliberate cultural choice. STAMP/CAST is a mature framework in safety engineering — and one of several systemic accident models (FRAM and AcciMap are alternatives) — but its application to AI systems is early, with the first structured attempts (e.g. STPA applied to frontier-AI development, arXiv:2506.01782, 2025) still exploratory. The transferable core: separate factual from analytical, publish the facts first, keep causes plural, and treat missing coordination as a real cause — not any specific regulatory apparatus.

Reusable procedure: the full AI incident investigation

1. Staff — four roles, separated, named on day zero.
2. Assemble — custodian’s manifest with fidelity labels before theory.
3. Isolate — H1/H2/H3 by record signature, ≥3 trials, verbatim citation.
4. Repair + calibrate — one variable per repair, bidirectional tests, calibrated clauses, joint ledgers.
5. Remediate + publish — scope from frozen records, packet with labeled inference and bounded generality.

Failure modes

• Role collapse. One head, all duties. Self-certifying timelines.
• Manifest skipping. Theorizing before freezing. Evidence shaped to fit the theory.
• Blame-first routing. Assignment before separation. Investigations of innocence, findings discounted.
• Promise prevention. “We will add checks” without versions, sets, or CI links. Prose as guardrail.
• Unbounded schedule. No definition of done. Perfectionism starving the conveyor.
• Generality smuggling. “Never again” in the packet. The forbidden claim; residuals exist, name them.
• Silent remediation. Accounts fixed, nothing published. Trust unbilled, regulators unmoved.
• Multi-variable incident repair. Edges, prompts, and thresholds in one deploy. Attribution destroyed at scale.

Limits, per contract: one packet covers one incident under one evidence manifest; it certifies this class on these edges under these clauses — never immunity, never permanence across regimes, never unknown classes. UNKNOWN wherever fidelity was reconstructed rather than frozen.

References

• National Transportation Safety Board. The Investigative Process. https://www.ntsb.gov/investigations/process/Pages/default.aspx (party system; factual report vs. analysis vs. probable cause). See 49 CFR § 831.4(c) — investigations not conducted to determine rights, liabilities, or blame — and 49 U.S.C. § 1154(b) — Board reports inadmissible in damages suits.
• Nancy Leveson. Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press, 2011. https://doi.org/10.7551/mitpress/8179.001.0001 (STAMP accident model; safety constraints and control structure).
• Nancy Leveson. CAST Handbook: How to Learn More from Incidents and Accidents. MIT PSAS, 2019. http://sunnyday.mit.edu/CAST-Handbook.pdf
• John Allspaw. Blameless PostMortems and a Just Culture. Etsy Code as Craft, 2012. (Cross-ref Ch 56 — carried into the packet’s evidence rules.)
• Richard Cook. How Complex Systems Fail. University of Chicago, 2000. (Cross-ref Ch 56 — root-cause-as-artifact; why contributing factors stay a list.)

Debugging Checklist

• Four roles staffed and separated (roster published)?
• Evidence manifest frozen before any theorizing (fidelity labeled)?
• H1/H2/H3 pre-written with distinct boundary/join signatures?
• All cells run ≥3 trials with distributions recorded?
• Every verdict cites its deciding record verbatim?
• Each repair verified bidirectional (red pre, green post)?
• Clauses registered with calibration sets + thresholds as setup choices?
• Joint ledger (Q/L/S) booked per repair?
• Remediation scoped from frozen records (not estimates)?
• Packet published with labeled inference, named UNKNOWNs, bounded generality?
• No narrative, confidence, agreement, single runs, or calm cited as evidence?

What This Chapter Established

• The full AI incident investigation: four roles, five phases, one published packet — demonstrated on the constructed weekend duplicate-refund incident, no measured runs claimed.
• The contract/consumption/coordination separation (H1/H2/H3) with verbatim citation and the manifest-before-theory ordering rule.
• Lab 59 as a proposed packet drill the reader executes; the Full Incident Investigation Checklist contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any cross-incident immunity, any threshold permanence, or any unknown-class completeness. One incident fully accounted; nothing universal.
• Position in the arc: Chapters 57–58 compress the method to ten and sixty minutes with honest skips; this chapter spends the full budget with nothing skipped but generality. The playbook’s deepest procedure, bounded on its final page.
• Research grounding: the NTSB template (party system, factual report before analysis, probable cause + separate contributing factors, investigations not conducted to determine rights/liabilities/blame — 49 CFR § 831.4(c)) and systemic causation (Leveson STAMP/CAST, one of several systemic models) as the basis for H3 as a first-class control-structure cause; the published packet is the Chapter 45 diagnostic-case record at its last stage.

Next

Three budgets, one method — triage in ten, isolate in sixty, account fully in days. Each procedure consumed the instruments of Parts I–IX without restating them, which raises the book’s final question: what exactly has the practitioner accumulated across sixty chapters, how is it selected under pressure, and what did it actually prove? Chapter 60, “The Debugging AI Toolkit,” inventories the whole kit with a selection guide; what the book established — and what it left provisional — is its chapter’s to establish, not this one’s.