Part X — The Debugging AI Playbook

Ten minutes, one failing request, no permission to boil the ocean

Part IX closed with the full method under production load: emit, convey, enforce, book, respond in order. Now the practitioner gets reality: ten minutes before the deploy review, one user-flagged refusal that “worked yesterday,” and no authority to freeze slices or convene commanders. The temptation is full rigor in miniature — a five-stage conveyor, a calibration set, and a ledger before lunch. That fiction produces the worst outcome: ten minutes spent starting everything and finishing nothing, with a confident verdict announced to fill the silence.

OBSERVATION: one flagged request (output text + timestamp only); no bundle, no baseline access, ten-minute hard stop. HYPOTHESIS H1 (input-local): the failure follows this input’s assembly (retrieval, truncation, parameters). H2 (environment drift): the input is innocent; model, index, or config moved underneath. H3 (triage-only): ten minutes cannot separate H1 from H2 — the honest output is a routed handoff, not a verdict. INFERENCE: none yet — and the chapter’s first discipline is admitting that H3 may be the final answer at this budget.

This chapter’s question: what does the compressed procedure establish in ten minutes — and what does it honestly refuse to claim?

Why “quick full diagnosis” fails first

The obvious move — running the whole method faster — fails because rigor does not compress; only scope does. Five fictions hide behind ten-minute verdicts:

1. Single-run conviction. One retry “fixes it,” verdict announced. Nondeterminism requires trials; ten minutes rarely holds three — so the verdict is UNKNOWN, not green.
2. Explanation-as-trace. Asking the model why it refused and citing the answer. Self-narration is behavior, not trajectory — forbidden at every budget, cheapest here.
3. Scoreboard diagnosis. A retrieved chunk scored 0.91, therefore correct. Correlation is not causation at any depth; at ten minutes it is pure decoration.
4. Multi-fix flailing. Prompt tweaked, temperature lowered, chunk raised — all before the review. Attribution destroyed faster than at any other budget.
5. Handoff-free closure. “Probably the retriever” said aloud, nothing written, no artifact queued. Ten minutes that produce no handoff produce nothing.

OPINION: ten minutes buys a routed handoff or it buys damage. There is no third purchase.

The mental model: the ten-minute debug as triage, not diagnosis — freeze what you can, separate H1 from H2 with one cheap probe, and emit a handoff record that the next budget can consume. KEPT at this budget: one failing case specified, one cheap single-variable probe with a pre-written prediction, OBSERVED-vs-INFERRED labeling, and a written handoff. SKIPPED honestly: environment bisection (no pins available in ten minutes — H2 stays live unless a probe kills it), repeated trials beyond two (stated as a limit, not faked), calibration, ledgers, guardrail changes, and all causal claims.

Two of Agans’s nine debugging rules survive compression intact and one is explicitly the point here. “Quit thinking and look” — get data before theorizing — is the one-probe step; ten minutes has no budget for a theory you did not test. “Change one thing at a time” is why the probe touches exactly one variable. And “if you didn’t fix it, it ain’t fixed” (Agans, Debugging, 2002) is the reason a single passing re-run is UNKNOWN, not green — the failure that stops on its own was not fixed by you, and may not have stopped. Triage keeps the rules that are cheap and drops the rules (“divide and conquer” across environment versions, “keep an audit trail” of many trials) that need time this budget does not have.

The method: the ten-minute procedure with explicit skips

Run six steps against the clock; each step has a skip rule:

1. Minute 0–1: specify one case. Write the failing input, the expected property, and the observed divergence in one sentence each. No case sentence, no procedure — vague unease is not debuggable at any budget.
2. Minute 1–3: freeze what you can. Copy the exact input text, output text, timestamps, and whatever pins are visible (model name as displayed, index date if shown). Hash nothing fancy — a pasted record with named UNKNOWNs beats a delayed perfect bundle. Label every missing field UNKNOWN.
3. Minute 3–6: one cheap probe. Pick the single highest-leverage single-variable probe available without tooling: re-run the identical input once (nondeterminism check), or re-run with retrieved context shown vs. hidden (retrieval-dependence check), or re-run with history truncated (assembly check). Pre-write the prediction in one line before running. One probe only — the second probe belongs to the hour.
4. Minute 6–8: read the probe honestly. Probe confirms H1-shaped → route to the Chapters 30–33 owner with the record. Probe confirms H2-shaped → route to environment/drift queue. Probe ambiguous or single-run → H3 stands: triage-only, handoff queued, no verdict. Agreement with your prior is not confirmation; the prediction line decides.
5. Minute 8–10: write the handoff. Fill the paper form below. Queue it to the named next owner or to your own hour-investigation slot. A handoff with UNKNOWNs named is success; a verdict without trials is failure.
6. Stop. Ship no prompt edit, no config change, no guardrail tweak from a ten-minute session. The rule is absolute: ten minutes diagnoses nothing shippable.

    flowchart TD
    S1["0-1 min: specify ONE case — failing input, expected property, observed divergence"] --> S2["1-3 min: freeze what you can — input / output text, visible pins; name every missing field UNKNOWN"]
    S2 --> S3["3-6 min: ONE cheap single-variable probe, prediction written in one line BEFORE running"]
    S3 --> S4{"6-8 min: read the probe against the prediction, not the prior"}
    S4 -->|"identical re-run reproduces"| H1["H1 input-local -> route to the Ch30-33 owner"]
    S4 -->|"identical re-run diverges"| H2["H2 environment drift -> route to the drift queue"]
    S4 -->|"ambiguous / only 1-2 runs"| H3["H3 TRIAGE-ONLY — a valid verdict at this budget"]
    H1 --> S5["8-10 min: write the handoff record, route to a named owner / the one-hour slot"]
    H2 --> S5
    H3 --> S5
    S5 --> STOP["STOP — ship no prompt edit, no config change, no guardrail tweak from ten minutes"]
  

TEN-MINUTE RECORD (one case, one probe):
case: input ___ expected ___ observed ___ (first divergence: ___)
frozen: in/out text ___ pins visible ___ UNKNOWN: ___ (named)
probe: ___ predicted ___ OBSERVED ___ (runs: 1-2, insufficient for verdict — stated)
H1/H2/H3: live / exonerated / TRIAGE-ONLY  ROUTED TO: ___ (owner ___)
KEPT: case + probe + handoff | SKIPPED: bisection, trials x3, calibration, ledger, all fixes
RULE: no ship from ten minutes. Ever.

OBSERVATION (constructed illustration, not a measured run): the flagged refusal re-run identically answers correctly once — a single data point, honestly insufficient; the hidden-context probe is skipped for lack of tooling and named as such. UPDATED BELIEF: H1 stays live (possible transient assembly), H2 stays live (possible drift), H3 is the verdict — triage-only handoff to the one-hour queue with the freeze attached. Two hypotheses alive and one honest record beats one dead-certain fiction.

No model self-explanation, no confidence value, no chunk score, no single-run recovery, no agreement (“support agrees it’s the retriever”), and no downstream symptom (“no more flags today”) converts a ten-minute session into a diagnosis. The probe record routes; everything else waits for the hour.

Example: the refused refund, triaged before the review

The practitioner has nine minutes and one flagged refusal:

# ten-minute triage: specify, freeze, one probe, hand off (no repair branch exists)
case = specify(input_text, expected="cites policy section 4.2", observed="refused")  # 1 min
frozen = freeze_available(case)  # OBSERVATION + named UNKNOWNs (pins, index, params) — 2 min
prediction = "H1-shaped: identical re-run refuses again -> input-local; answers -> transient/drift"
result = rerun_identical(case)  # ONE probe, prediction pre-written — 3 min
handoff(queue="one-hour", record=(case, frozen, prediction, result))  # 2 min; stop.

In the constructed case the identical re-run answers correctly — consistent with transient assembly or drift, probative of neither. The handoff records exactly that plus the UNKNOWN pins, routes to the one-hour queue, and the practitioner walks into the review saying “triaged, routed, no verdict” instead of “probably fixed.” The licensed claim covers this triage of this case — not refusal diagnosis in general.

Lab 57: ten-minute drill with pre-written routing predictions (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own timed drill.

Setup. Collect three real flagged cases from your own traffic (or staged equivalents). Set a visible ten-minute timer per case. The procedure (habitual rushing vs. this chapter’s six steps) is the independent variable; cases and tooling are controlled.

Task.

1. Before each timer starts, write H1/H2/H3 in one line each with the single probe that would separate them and its predicted signatures.
2. Run the timer; execute specify → freeze → one probe → handoff; record per-minute timestamps verbatim.
3. Grade each session: handoff consumable by a fresh reader (could they run the hour from your record alone?) vs. verdict-announced vs. fix-shipped.

Success criterion. Three timed records, each with a case sentence, a freeze with named UNKNOWNs, one prediction line, one probe outcome, and a routed handoff — completed inside ten minutes each. A verdict or a shipped edit is explicitly scored as failure.

Companion tool: Ten-Minute Debug Checklist

What it accepts: the failing case sentence, the available-evidence freeze with named UNKNOWNs, the single probe definition with its pre-written prediction, and the probe outcome. What it performs: it verifies a case was specified before probing, exactly one variable was touched, the prediction preceded the run, the outcome is read against the prediction (not the prior), a handoff names the next owner, and no repair shipped — refusing to certify any session that announces verdicts on 1–2 runs. What it can establish: whether the session produced a consumable triage handoff — for the examined case only. What it cannot establish: causes, input-vs-environment separation (unless the probe cleanly kills one hypothesis — rare, and labeled as provisional), or anything shippable. It never treats self-explanations, scores, single runs, agreement, or quiet as session output. How its output changes your next action: handoff-consumable routes to the hour queue or named owner; probe-decisive (rare) routes to the corresponding chapter with provisional labeling; verdict-shaped routes to retraction and re-triage — each as one honest record.

Paper form, sufficient for this chapter:

Case: ___ diverged at ___ | Frozen: ___ UNKNOWN: ___
Probe: ___ predicted ___ observed ___ (runs ___, verdict INSUFFICIENT unless ___)
H1/H2/H3: ___ / ___ / ___  ROUTED TO: ___  SHIPPED: nothing (confirm ___)
KEPT: ___  SKIPPED (named): ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Triage, then stop.

Research lineage: triage vs. diagnosis, expert speed, and its failure mode

Triage is a discipline, not a rushed diagnosis. Medical triage and diagnosis are separate acts with separate outputs: triage sorts and routes under a time constraint; diagnosis names a cause with the time and tests to support it. This chapter’s H3 (“triage-only is a valid verdict”) is that distinction imported — the ten-minute output is a routing decision, and announcing a cause instead is a category error, not a faster diagnosis.

Experts really are fast — by pattern-matching, not analysis. Klein’s Recognition-Primed Decision model (from the Naturalistic Decision Making tradition; Sources of Power, 1998) found that fireground commanders, ER nurses, and other time-pressured experts do not enumerate and compare options — they recognize the situation as a known prototype, generate one candidate action, mentally simulate it, and act. This is what a skilled practitioner’s ten minutes actually looks like, and the chapter’s “one cheap probe with a pre-written prediction” is a deliberate slowing of exactly the risky step: the mental simulation is written down before the probe so the result can contradict it.

RPD’s failure mode is why ten minutes cannot certify. RPD is reliable only when the situation genuinely matches a prototype the expert has seen; on novel situations, pattern-matching produces a confident wrong recognition, and the expert does not notice the mismatch. AI failures are frequently novel (new prompt, new model version, new retrieval corpus), so the ten-minute recognition is exactly the case where RPD is least trustworthy — hence the absolute no-ship rule and the requirement that H2 (environment drift) stays live unless a probe kills it.

The written prediction is a frame made falsifiable. Klein, Moon, and Hoffman’s data/frame model of sensemaking (Making Sense of Sensemaking 2, IEEE Intelligent Systems, 2006) describes how people read a situation: a frame — a working account of what is going on — governs which data you notice, and as data arrives the frame is either elaborated (strengthened, often past what it has earned) or reframed (discarded for a better one). The default failure is silent elaboration: an implicit frame quietly absorbs disconfirming data and nobody registers the mismatch. Writing the probe’s prediction down before running it is a deliberate anti-elaboration move — it fixes the frame in a form a contradicting result can visibly break. That is why the ten-minute procedure spends a scarce minute on the prediction line and not on a second probe.

Time pressure degrades debugging specifically. Studies of developers under deadline pressure (cf. Parnin & Orso on how little automated tools help in realistic conditions) show the first casualties are systematic search and hypothesis tracking — precisely the things the paper form re-imposes by making them one line each.

Bounds: Agans’s rules are practitioner heuristics, not empirically ranked; RPD and the data/frame model are descriptive models from field studies, not prescriptions; the triage/diagnosis analogy is structural, not a claim that debugging is medicine. The transferable core: keep the cheap rules (look before theorizing, one variable, distrust a failure that stopped itself), name the skipped ones, and treat fast recognition as a routing hypothesis to be written down and tested, never as a verdict.

Reusable procedure: the ten-minute debug

1. Specify — one case, one expected property, one observed divergence (1 min).
2. Freeze — available evidence pasted, UNKNOWNs named (2 min).
3. Probe once — single variable, prediction pre-written (3 min).
4. Read honestly — prediction decides; H3-triage is a valid verdict (2 min).
5. Hand off — named owner, consumable record (2 min).
6. Ship nothing — no edits from ten minutes, no exceptions.

Failure modes

• Verdict theater. Announcing causes on one run to fill silence. Silence with a handoff beats certainty without trials.
• Explanation-as-trace. Citing the model’s apology as the diagnosis. Forbidden everywhere; fatal here.
• Scoreboard triage. Chunk scores or confidence cited as routers. Numbers without causation.
• Probe stacking. Three “quick” probes in ten minutes, none predicted. Motion disguised as method.
• Live repair. Shipping a tweak from triage. The absolute prohibition; the source of most second incidents.
• Handoff-free sessions. Ten minutes, zero writing. Nothing happened that the next budget can use.
• H3 shame. Treating triage-only as failure and upgrading it to verdict. H3 honestly held is this chapter’s success state.

Limits, per contract: one session covers one case with 1–2 runs; it establishes routing only, never causation; H2 survives unless killed by a probe; everything shippable waits for deeper budgets. UNKNOWN wherever pins, trials, or tooling were absent — which at ten minutes is most places, honestly named.

References

• David Agans. Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems. AMACOM, 2002. (Rules cited: “Quit Thinking and Look,” “Change One Thing at a Time,” “Keep an Audit Trail,” “If You Didn’t Fix It, It Ain’t Fixed.”)
• Gary Klein. Sources of Power: How People Make Decisions. MIT Press, 1998. (Recognition-Primed Decision model; Naturalistic Decision Making.)
• Gary Klein, Brian Moon, and Robert R. Hoffman. Making Sense of Sensemaking 2: A Macrocognitive Model. IEEE Intelligent Systems 21(5), 2006, pp. 88–92. https://doi.org/10.1109/MIS.2006.100 (The data/frame model: frames are elaborated or reframed as data arrives; the written prediction fixes the frame so a probe can break it.)
• Gary Klein, Roberta Calderwood, and Anne Clinton-Cirocco. Rapid Decision Making on the Fire Ground. Journal of Cognitive Engineering and Decision Making, 1986/2010. https://doi.org/10.1518/155534310X12844000801203
• Chris Parnin and Alessandro Orso. Are Automated Debugging Techniques Actually Helping Programmers? Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), 2011. https://doi.org/10.1145/2001420.2001445 (developer behavior under realistic constraints; cross-ref Ch 1, Ch 17).
• Herbert Simon. Rational Choice and the Structure of the Environment. Psychological Review, 63(2), 1956. (Satisficing / bounded rationality — the theoretical frame for “route, don’t optimize” at a budget.)

Debugging Checklist

• One case specified (input, expected, observed) before any act?
• Available evidence frozen with UNKNOWNs named?
• H1/H2/H3 written with one probe’s predicted signatures?
• Exactly one single-variable probe run, prediction-first?
• Outcome read against the prediction (not the prior)?
• Handoff written, routed to a named owner or hour queue?
• Nothing shipped (confirm: zero edits)?
• KEPT vs. SKIPPED stated on the record?
• No self-explanation, score, single-run verdict, agreement, or quiet cited?

What This Chapter Established

• The ten-minute debug as honest triage: specify, freeze, one predicted probe, labeled handoff, absolute no-ship rule — demonstrated on the constructed flagged refusal, no measured runs claimed.
• The KEPT-vs-SKIPPED scoping contract: routing is the product; bisection, trials, calibration, ledgers, and fixes are explicitly deferred, not compressed.
• Lab 57 as a proposed timed drill the reader executes; the Ten-Minute Debug Checklist contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any diagnostic capability beyond routing, any input-vs-environment separation as a rule, or any shippable outcome. Ten minutes triages; nothing more.
• Position in the arc: Part IX built the full-depth method; this chapter opens Part X by stating what survives contact with the calendar. Triage with integrity.
• Research grounding: triage-vs-diagnosis as separate acts; Agans’s compressible rules (“quit thinking and look,” “change one thing,” “if you didn’t fix it, it ain’t fixed”); Klein’s RPD for expert speed and its novel-situation failure mode as the reason ten minutes cannot certify; the data/frame model (Klein, Moon & Hoffman) for why the probe’s prediction is written down first — to make the frame breakable instead of silently elaborated.

Next

Ten minutes routes; it never resolves. The natural next budget — an hour of protected focus with pins and history available — restores bisection, trials, and minimization. But the hour has its own honest boundary: it isolates, it does not certify. Chapter 58, “The One-Hour Investigation,” states the standard-depth procedure; what an hour can close vs. what it must still hand upward is its chapter’s to establish, not this one’s.