Part VIII — Building the AI Debugger

The leaderboard that measured nothing

Chapter 49 verified single diagnoses; this chapter asks what would measure debugging skill itself. A team ships “DebugScore”: an assistant is given fifty broken AI setups, its fixes are graded by whether the symptom disappears, and the resulting number — 87% — is cited in planning docs as proof the debugger works. Nobody asks whether the tasks resemble the book’s failure modes, whether grading checked exoneration of alternatives or just relief, whether the assistant had seen the tasks in training, or whether one trial per task decided each score. The number is precise, reported, and evidentially empty.

OBSERVATION: fifty tasks graded on symptom relief, one trial each, task provenance undocumented, grading rubric relief-only, no contamination analysis. HYPOTHESIS H1 (skill): the score measures debugging capability. H2 (memorization): the score measures prior exposure to the tasks. H3 (relief-gaming): the score measures symptom-masking without diagnosis. INFERENCE: none yet — H1/H2/H3 predict different outcomes under rubric and contamination controls: H1 survives ledger-graded rubrics on held-out tasks; H2 collapses on novel tasks; H3 collapses when grading requires exoneration ledgers rather than relief.

This chapter’s question: how do we design a benchmark for debugging skill — tasks, grading, contamination control — that deserves the name, while reporting no scores because none are claimed here?

Why “fix-rate on collected bugs” fails first

The obvious move — gather bugs, measure fix-rate — fails because debugging skill is not fix-rate. Five defects hide behind relief-graded leaderboards:

1. Relief grading. Credit for symptom disappearance rewards masking (H3) and punishes honest PROVISIONALs. Grading must score the ledger — predictions, exonerations, independence, reversal-or-cap — not the quiet symptom. There is already a debugging benchmark: Tian and colleagues’ DebugBench has 4,253 code-repair instances across bug categories in three languages, built with explicit attention to data-leakage risk (Tian et al., 2024). But it grades on pass-rate — did the repaired code pass the tests — which is relief grading. Scoring the process instead, as Lightman and colleagues showed for reasoning (“Let’s Verify Step by Step” — supervising each step beats supervising only the answer — Lightman et al., 2023), is what the ledger rubric does.
2. Unmapped tasks. Fifty RAG bugs and zero agent-trajectory or multi-agent-boundary tasks. An unmapped task set certifies nothing about uncovered layers; tasks must map to the book’s chapters (failure families), or coverage is unknown.
3. Single-trial scoring. One run per task on nondeterministic systems. Scores without trial discipline measure luck with error bars nobody computed.
4. Contamination blindness. Tasks drawn from public issue trackers the assistant trained on. Without provenance tracking and held-out/novel-task arms, H2 (memorization) is live for every high score.
5. Reported-but-unearned numbers. Publishing a score the design cannot support. This chapter’s explicit NON-claim: no benchmark scores are reported here — no runs were executed, no tasks administered, no numbers exist to report.

OPINION: a debugging benchmark graded on relief is a fire department graded on smoke complaints — it incentivizes opening windows, not extinguishing fires.

The mental model: the benchmark is an exam with an answer key made of ledgers. Each task ships with its crash-dump bundle, its hypothesis space, and its grading rubric (the Ch49 ledger as answer key: which predictions must match, which exonerations must be cited, what independence and trial counts are required). The score measures ledger quality under controlled conditions — not relief speed, not fluency, not confidence calibration.

The method: task families, ledger rubrics, anti-contamination

AIDebugBench as designed (not run) has three components:

1. Task families mapped to book chapters. Each family names its chapter and failure mode: values/state localization (Ch01–05 style), distribution/nondeterminism (Ch-part on training systems), evidence/RAG faults (stale vs. demotion vs. override), trajectory faults (loops, tool misuse), boundary faults (handoff/protocol/coordination per Ch43), and experimental-causality tasks (Ch47–49 ledger completion). Every task ships a frozen bundle (Ch45 manifest), a stated intent, and ≥2 plausible hypotheses — never single-story tasks. SWE-bench Verified is the evidence that confirming each task is well-formed and its intended diagnosis is reachable from the bundle is a large, necessary effort — 93 developers, three independent reviews each, over 1,699 candidate problems to reach 500 clean ones (OpenAI, 2024) — not a formality; every item needs an independent well-formedness review before it enters the pool.
2. Ledger rubrics, not relief rubrics. Points awarded for: correct first-divergence pointer with cited records; enumerated space covering the task’s layers; divergent pre-dated predictions; single-variable runs with required trial counts; losers exonerated with deciding runs cited; independence stream present; reversal or stated cap; human-sign-off simulation (flagging high-impact actions). Zero points for relief without ledger; negative marking for forbidden evidence (explanation/scores/agreement/single-run/symptom cited as proof).
3. Anti-contamination controls. Task provenance recorded per item (seeded-novel vs. adapted-public, with dates); a held-out arm never published in training-accessible form; performance reported split by provenance (novel vs. adapted) so memorization shows as a gap; task rotation policy (retire items showing exposure signatures); and a ban on training on the held-out arm stated in the release terms. Any score reported without the provenance split is graded INADMISSIBLE by this design.

    flowchart TD
    D["AIDebugBench design (no administrations)"] --> TF["task families mapped to book chapters; uncovered families declared, not hidden"]
    TF --> TK["each task ships: frozen bundle manifest + stated intent + >=2 plausible hypotheses"]
    TK --> WF["independent well-formedness review per item (intended diagnosis reachable from the bundle)"]
    WF --> RB["ledger rubric: divergence cited / space covered / predictions pre-dated / single-variable / trials met / losers exonerated / independence / reversal-or-cap"]
    RB --> PN["relief alone = 0; forbidden-source citation = negative marking"]
    PN --> AC["anti-contamination: provenance per item, held-out arm, novel-vs-adapted split reporting, rotation of exposed items"]
    AC --> RP["report per-family profiles as distributions with spread — never a lone headline number"]
  

BENCH DESIGN SKETCH (design only; constructed, never administered):
Families: [localize×n] [evidence×n] [trajectory×n] [boundary×n] [ledger×n]
  — each task: bundle manifest + intent + ≥2 plausible H + chapter map.
Rubric per task (ledger points, not relief):
  divergence cited __ | space covered __ | predictions pre-dated __
  single-variable __ | trials met __ | losers exonerated __
  independence __ | reversal-or-cap __ | impact flagged __
  forbidden-source penalty __ | TOTAL __ (relief alone = 0)
Controls: provenance per item (novel/adapted+date) | held-out arm __
  rotation rule __ | training-on-held-out banned __
SCORES REPORTED IN THIS CHAPTER: none. No tasks administered,
no runs executed. Any number attributed to AIDebugBench from this
text would be fabrication.

OBSERVATION (design artifact, not a measured run): the sketch above specifies gradable structure with no administrations. UPDATED BELIEF: no update about any debugger’s skill is licensed — H1/H2/H3 about any system remain fully live; the design is complete, the evidence is zero, and that sentence is the point.

No vendor accuracy claim (attribute and date any cited claim; model facts are changeable), no paper’s reported number repeated as fact, no internal pilot score smuggled in as illustration. This chapter contains zero benchmark numbers by construction.

Example: grading one task two ways

Take the stale-snapshot task from Chapters 47–49. Relief grading: “citation returned after re-index → 1 point.” Ledger grading: divergence pointer with ranked-list cite (required), H1/H2/H3 space with signatures (required), pre-dated P1–P5 (required), single-variable series with ×3 trials (required), H2/H3 exonerations cited (required), independence stream (required), reversal or cap (required). An assistant that re-indexes first and gets relief scores 1/1 on relief grading and fails ledger grading (predictions post-dated, no exonerations, no independence) — exactly the H3 relief-gaming the design exists to catch. The example is a grading comparison on paper; no assistant was run.

Research lineage: the contamination control is a solved design problem

GSM1k is one template for the held-out arm. Zhang and colleagues rebuilt a grade-school-arithmetic test set from scratch — human-authored, no LLM assistance, matched to the original’s style and difficulty — and found several model families dropping 8–13 points on the fresh version, with the drop correlated with how likely the model was to reproduce the original benchmark’s items (Zhang et al., 2024). Their two design moves are AIDebugBench’s anti-contamination controls: a parallel-difficulty novel arm, and a held-out release policy. “Report performance split by provenance” is their memorization signal, generalized. A second strategy comes from LiveCodeBench and LiveBench: stamp every task with a release date and score a model only on tasks released after its training cutoff, so contamination shows up directly as a drop on pre-cutoff items (Jain et al., 2025). AIDebugBench can run both — a static held-out arm and a rolling arm of freshly-seeded tasks reported by vintage.

Two calibrations keep the provenance split honest. A modest novel-vs-adapted gap is expected: rebuilding a test set to the same protocol produces a 3–15% intrinsic drop that is not leakage (Chapter 16, Recht et al.), so only a gap clearly beyond that band — or one correlated with item-reproduction likelihood, GSM1k’s method — convicts memorization. And the held-out arm guards mainly against training-set contamination: a community slowly overfitting the public arm through repeated tuning (adaptive overfitting) is, on clean large splits, empirically mild (Chapter 16, Roelofs et al.).

Construct validity is the “unmapped tasks” problem. Raji and colleagues argue that benchmarks routinely claim to measure a general capability while testing a narrow slice, and that the mismatch is rarely examined (Raji et al., 2021). Fifty RAG bugs graded well tell you about RAG-bug fixing, not “debugging skill.” The chapter-mapped task families with declared gaps are the response: the benchmark states exactly which slice it covers.

Ledger grading is process supervision. The move from “did the symptom go away” to “did the diagnosis follow the method” is the outcome-vs-process distinction that process reward models formalize. It is harder to grade and harder to game — which is the trade the chapter makes deliberately.

Lab 50: grade a debugger output with the ledger rubric (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own grading record.

Setup. Take one debugging transcript (yours, a colleague’s with permission, or an assistant’s on a bundled failure). The grading regime (relief-only vs. this chapter’s ledger rubric) is the independent variable; the transcript is controlled.

Task.

1. Before grading, write H1/H2/H3 with distinct predicted grading outcomes: H1: “ledger rubric downgrades a relief-passing transcript (gaming exposed)”; H2: “both regimes agree (transcript genuinely complete or genuinely poor)”; H3: “ledger rubric passes a relief-failing transcript (honest PROVISIONAL credited).”
2. Grade twice, row by row; mark provenance of the task (novel/adapted/unknown) and trial counts.
3. Record the score gap and which rubric rows decided it.

Success criterion. Two completed grading sheets with per-row points, provenance marked, and the deciding rows named. A single number without per-row evidence is explicitly not completion.

Companion tool: Benchmark Runner

What it accepts: the task set with manifests and chapter maps, the ledger rubric version, the provenance log, and the subject’s transcripts with trial logs. What it performs: it administers tasks blind (no solutions in context), enforces trial counts, grades per-row against the ledger answer key, applies forbidden-source penalties, splits results by provenance (novel vs. adapted), checks exposure signatures, and emits per-family profiles — never a single headline number alone. What it can establish: how a subject’s debugging process scores per family under these controls — for administered subjects only, when runs actually occur. What it cannot establish: anything, until administrations happen — and this chapter administers none. When run, it still cannot establish generality beyond the task families, immunity to future contamination, or production competence. It never treats relief, fluency, confidence, agreement, or single trials as skill. How its output changes your next action: profile emitted → weakest family routes to targeted practice (its chapters); provenance gap large → distrust adapted-arm scores, weight novel-arm; exposure signatures → retire tasks, rotate.

Paper form, sufficient for this chapter:

Subject ___ (system ___ v___, date ___) Rubric v___ Tasks (n=___)
FAMILIES: localize __/__ | evidence __/__ | trajectory __/__ | boundary __/__ | ledger __/__
PROVENANCE SPLIT: novel __/__ vs adapted __/__ (gap ___)
TRIALS MET: __/__ tasks | FORBIDDEN penalties: ___ | EXPOSURE flags: ___
ADMINISTRATIONS IN THIS CHAPTER: 0 — form shown for future use only.

Where a software implementation does not yet exist in the reader’s stack, this record is the tool — to be used, not merely admired.

Reusable procedure: build benchmarks that deserve trust

1. Map tasks to chapters — every item names its failure family; uncovered families are declared, not hidden.
2. Ship bundles — frozen manifests with intent and ≥2 plausible hypotheses per task.
3. Grade ledgers — rubric rows from Ch49; relief alone scores zero.
4. Control contamination — provenance per item, held-out arm, split reporting, rotation.
5. Report profiles as distributions — each per-family score is a distribution over the required trials and multiple task seeds, reported with its spread (Ch12, Bouthillier et al.); a profile difference inside that spread is noise. Exposure flags attached; no naked headline numbers.

Failure modes

• Relief leaderboards. Fix-rate as skill. Relief is gameable; ledgers are auditable.
• Unmapped coverage. Task counts without family maps. Fifty of one kind is one data point wearing a crowd costume.
• Single-trial scores. Luck reported as capability. Trial discipline or silence.
• Provenance amnesia. Scores without novel/adapted splits. Unexamined H2 (memorization) voids H1 (skill).
• Headline-number reporting. One number, no profile. Profiles route practice; headlines route press.
• Citing this chapter’s non-scores. Any number attributed to runs described here. NON-claim restated: no administrations, no scores, no exceptions.

Limits, per contract: the design warrants future administrations, not present conclusions; any built instance covers its task families under its rubric/provenance regime only; version or task-pool changes bound old scores until re-administered. UNKNOWN wherever provenance is unrecorded or trials undersupplied.

References

• Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Zhiyuan Liu, and Maosong Sun. DebugBench: Evaluating Debugging Capability of Large Language Models. Findings of the Association for Computational Linguistics: ACL, 2024, pp. 4173–4198. https://aclanthology.org/2024.findings-acl.247/
• Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, et al. A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k). NeurIPS Datasets and Benchmarks Track, 2024. https://arxiv.org/abs/2405.00332
• Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the Everything in the Whole Wide World Benchmark. NeurIPS Datasets and Benchmarks Track, 2021. https://arxiv.org/abs/2111.15366
• Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step. International Conference on Learning Representations (ICLR), 2024 (arXiv:2305.20050). https://arxiv.org/abs/2305.20050
• Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. International Conference on Learning Representations (ICLR), 2025 (arXiv:2403.07974). https://arxiv.org/abs/2403.07974
• OpenAI. Introducing SWE-bench Verified. 2024. https://openai.com/index/introducing-swe-bench-verified/

Debugging Checklist

• Tasks mapped to book chapters/failure families with gaps declared?
• Every task ships a frozen bundle, intent, and ≥2 plausible hypotheses?
• Rubric grades ledger rows (relief alone = 0, forbidden sources penalized)?
• Trial counts enforced per task (≥3 where nondeterministic)?
• Provenance per item with a held-out arm and split reporting?
• Rotation/retirement policy for exposed items stated?
• Results reported as per-family profiles, never a lone headline?
• NON-claim honored: no scores reported from undesigned/undisclosed runs?

What This Chapter Established

• AIDebugBench as benchmark design: chapter-mapped task families, ledger (not relief) rubrics with forbidden-source penalties, and anti-contamination controls (provenance, held-out arm, split reporting, rotation) — design only, demonstrated on a paper grading comparison, no measured runs claimed.
• The exam-with-ledger-answer-key mental model and the per-family-profile reporting rule.
• Lab 50 as a proposed dual-regime grading record the reader executes; the Benchmark Runner contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: anything about any debugger’s skill — explicit NON-claim: zero administrations, zero scores reported, zero exceptions. One design finished; nothing measured.
• Research grounding: a debugging benchmark already exists (DebugBench) but grades on pass-rate = relief; ledger grading is process supervision (Lightman et al.), harder to grade and harder to game; the anti-contamination controls are GSM1k’s method (parallel-difficulty novel arm, held-out release, provenance-split reporting — Zhang et al.) plus LiveCodeBench’s time-windowed scoring (tasks stamped with release dates; score only post-cutoff items — Jain et al.), with the novel-arm drop calibrated against a 3–15% intrinsic re-sampling band and adaptive overfitting noted as the milder, separate worry (Recht et al.; Roelofs et al., Ch16); task well-formedness must be independently verified (SWE-bench Verified’s 93-developer effort); per-family scores are distributions, not points (Bouthillier et al., Ch12); and the chapter-mapped families with declared gaps address the construct-validity problem (Raji et al.).

Next

The benchmark can grade debuggers — but who grades the grader? Benchmarks, checkers, and dump-builders are software, and software has failure modes: stale bundles, broken probes, checker bugs that pass everything. A debugger that cannot suspect its own tooling inherits every Part VII–VIII discipline as theater. Chapter 51, “Debug the Debugger,” turns the book’s method on its own instruments — without borrowing the results of Part IX, which must earn its own claims.