Part IV — Debugging Models

Everybody blames the weights

Chapter 17 closed with five suspects hiding behind the same glass: prompt, retrieval, context, parameters, weights. The team’s reflex is to blame the last one — “the model can’t do split shipments, we need a bigger model” — and the vendor’s reflex is to agree, helpfully, in the direction of a larger invoice.

Concrete failure. The support bot answers a split-shipment refund wrong: it cites the general 30-day policy, ignores the section 4.2 exception sitting in the knowledge base, and computes the wrong amount. The ticket says “model failure.” That label is a HYPOTHESIS, not an OBSERVATION — and in this author’s experience it is wrong about half the time, because everything the weights did was downstream of everything the pipeline handed them.

OBSERVATION: frozen bundle reproduces the wrong citation at temperature 0 (MEASUREMENT, 5/5 trials, same revision). HYPOTHESIS H1 (pipeline): the assembled context never contained 4.2, or the instruction buried it — same weights decide correctly once handed the right input. HYPOTHESIS H2 (weights): the model cannot apply 4.2 even when shown it — the defect survives pipeline repair. HYPOTHESIS H3 (params): the sampling configuration (temperature, top-p, max-tokens cutoff) destroys a correct decision — the defect moves with parameters, not content. INFERENCE: none yet — the wrong paragraph is compatible with all three. Attribution requires swaps, not arguments.

This chapter’s question: before touching weights, prompts, or budgets — which layer owns this failure?

Why “try a bigger model” fails first

The obvious move — swapping to a larger or newer model and watching the answer improve — fails as diagnosis because it changes the suspect and the crime scene together. A bigger model tolerates worse retrieval; a newer revision carries a different system prompt; the demo passes and the pipeline rots underneath, now certified by a confounded experiment. Three attribution traps:

1. Downstream-symptom-as-cause. The answer is where the failure surfaces; the pipeline is often where it starts. Blaming the last component is like blaming the printer for a spreadsheet error.
2. Confounded swap. New model + new defaults + new template in one move. Improvement proves nothing except that something in the bundle changed.
3. Single-failure capability limit. One wrong answer does not bound what the model can do — capability claims need fixtures, not anecdotes. A single failure marks a case UNKNOWN for capability purposes, never a ceiling. Two measured effects make this worse: Liu and colleagues found a U-shaped position curve — models use information at the start and end of a long context but miss the same information in the middle (Liu et al., 2024) — and Sclar and colleagues found that semantically identical prompts in different formats (separator choice, casing, spacing) can swing a model’s accuracy by tens of points (Sclar et al., 2024). A failure that moves when you reorder the context or reformat the prompt was never a weights failure. And a capability claim from a pass/fail on one fixture is exactly the shape Schaeffer and colleagues showed can be a metric artifact: measured with a discontinuous metric, capability looks like a sudden cliff; measured with a continuous one, it improves smoothly and predictably (Schaeffer, Miranda & Koyejo, 2023). “The model can’t do X” needs a rate on a fixture — and the metric’s shape still matters after that.
4. Prompt-tweak loop. Editing wording for an hour without ever running the pipeline swap. Prompt iteration without attribution is search without a compass — motion billed as diagnosis.

OPINION: most “model failures” that reach practitioners are pipeline failures with model-shaped symptoms. The weights are the most expensive layer to change and therefore should be convicted last, not first. Triage exists to protect the budget from the reflex.

The mental model: the Chapter 4 stack, re-cut for model systems. Symptom → behavior → pipeline (prompt, retrieval, assembled context, parameters) → weights → intent. Each layer states a falsifiable contract; each is ruled out by a swap probe that holds everything else constant. The ordering principle is cost: rule out the cheap, reversible layers before the expensive, irreversible one.

The method: swap probes in stack order

Two complementary swaps, run in order, one variable per experiment.

Each probe is a trial series, not a single call — the ≥5 trials on the rows below is not caution, it is the floor of what a stochastic system will tell you. One output cannot separate a fixed model from a lucky draw. The stance here is only do not trust one run; Chapter 21 formalizes it, treating the distribution of outcomes as the debugging object and showing how to characterize and compare it.

1. Same-model × different-pipeline. Freeze the weights and revision; repair one pipeline element (hand the model the exact 4.2 text; strip the contradicting instruction; fix the truncation). Prediction if H1: the frozen model now answers correctly, ≥5 trials. The pipeline owned the failure; weights are exonerated for this fixture.
2. Same-pipeline × different-model. Only if probe 1 fails: freeze the exact input bytes and parameters; run a second model revision (or a second configuration) on the identical bundle. Prediction if H2: the failure follows the weights — model A fails where model B passes on byte-identical input. Prediction if H3: the failure follows the parameters — same weights pass at temperature 0 with adequate max-tokens and fail otherwise.
3. Intent check (Chapter 4’s bottom layer). If both swaps fail everywhere, quote the spec: does any policy line actually define the split-shipment amount? If not, the deliverable is a spec decision, and every model swap proposed before that moment was premature.

    flowchart TD
    BASE["baseline: frozen bundle, >=5 trials, counting criterion on the fixture"] --> P1["Probe 1: same weights + revision, repair ONE pipeline element (hand it 4.2 / strip the instruction / fix truncation)"]
    P1 --> D1{"failure flips to pass?"}
    D1 -->|yes| H1["H1 convicted: the pipeline owns it — stop, repair, add the fixture"]
    D1 -->|no| P2["Probe 2: byte-identical input + params, different model revision"]
    P2 --> D2{"failure follows the weights?"}
    D2 -->|yes| H2["H2 convicted: weights — route to signals / diffs (Ch22-23)"]
    D2 -->|no| P3["Probe 3: same bytes + weights, deterministic params + room to finish"]
    P3 --> D3{"failure flips with params?"}
    D3 -->|yes| H3["H3 convicted: sampling — route to Ch21"]
    D3 -->|no| INT["intent check: quote the spec line — if none defines the amount, it is a spec decision, not a model swap"]
  

# attribution swaps: one layer moves per experiment, >=5 trials each
base = freeze(model_rev="rev-A", input_bytes=assembled, params={"temperature": 0, "seed": 7})
print("baseline:", run(base, trials=5))                          # MEASUREMENT
# Probe 1 (H1): same model, repaired pipeline — hand it section 4.2 verbatim
print("probe-pipeline:", run(base.with_context(sections=["4.2-exception"]), trials=5))
# Probe 2 (H2), only if probe 1 holds the failure: same bytes, different weights
print("probe-weights:", run(base.with_model("rev-B"), trials=5))
# Probe 3 (H3): same bytes and weights, sampling moved toward deterministic + room to finish
# (temperature 0 narrows the distribution; it does not guarantee bitwise replay — Ch21)
print("probe-params:", run(base.with_params({"temperature": 0, "max_tokens": 2000}), trials=5))
# FORECASTs pre-written: H1 flips only probe-pipeline; H2 flips only probe-weights; H3 flips only probe-params.

OBSERVATION (constructed illustration, not a measured run): baseline 2/12; probe-pipeline 11/12; probe-weights and probe-params never ran — H1 convicted first and the stack stopped. UPDATED BELIEF: pipeline owns this fixture’s failure; weights exonerated for these cases under this revision. A weights defect elsewhere remains UNKNOWN, not disproved. INFERENCE: fix retrieval ranking + add the 4.2 regression fixture; no model change is justified by this incident.

Note the stop rule at work: probe 1 convicted, so probes 2 and 3 never ran. That restraint is load-bearing — every unrun probe is budget and attention preserved for the convicted layer’s repair. Had probe 1 sustained the failure (repaired context still 2/12), the same discipline would have demanded probe 2 on byte-identical input before any weights conclusion, because only identical bytes make the weights the remaining suspect.

Research lineage: the pipeline causes that wear a model’s face

Position is a pipeline variable that looks like a capability limit. The “lost in the middle” effect means that a RAG answer can be wrong purely because the decisive passage landed at rank 8 of 15 in the assembled context. The same model, handed the same passage first or last, answers correctly (Liu et al., 2024). Probe 1’s with_context(sections=["4.2-exception"]) implicitly fixes position as well as presence; a stricter version of the probe moves the passage to the top and re-runs. Hsieh and colleagues traced the effect to an intrinsic U-shaped attention bias — tokens at the start and end draw disproportionate attention regardless of relevance — and showed it persists even in models specifically trained on long contexts, but that a training-free inference-time calibration that subtracts the positional bias recovers up to about fifteen points of RAG accuracy (Hsieh et al., 2024). So a position-sensitive failure is doubly not a weights defect: the cause is known and mechanistic, and the fix lives outside the weights.

Format is a pipeline variable that looks like a capability limit. Sclar and colleagues’ FormatSpread shows that the choice of Q:/A: versus Question -/Answer -, or one newline versus two, changes measured accuracy by margins large enough to reverse model rankings (Sclar et al., 2024). A prompt-template change between two revisions is a confound exactly on the scale of the capability difference you think you are measuring.

The pipeline suspects are enumerable. Barnett and colleagues’ seven RAG failure points (Chapter 4) are the checklist for probe 1: missing content, top-ranked miss, relevant chunk not assembled, answer not extracted, wrong format, wrong specificity, incomplete (Barnett et al., 2024). Walk them before the weights swap.

Lab 18: convict the layer, cheapest first

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own swap table.

Setup. Take one “model failure” (or inject one: delete the decisive section from retrieval while keeping the model fixed). Freeze the bundle. Build a 10-case fixture with a counting criterion (correct citation + correct amount).

Task.

1. Write H1/H2/H3 with distinct numeric FORECASTs before swapping (e.g., “H1: repaired-pipeline ≥8/10, repaired-weights unnecessary; H2: repaired-pipeline ≤4/10 AND different-model ≥8/10 on identical bytes; H3: deterministic-params flip ≥8/10 with identical bytes and weights”).
2. Independent variable per run: exactly one swapped layer (pipeline element, or weights, or params). Controlled variables: everything else pinned, including seed and input bytes.
3. Run baseline + probe 1 (≥5 trials each). Run probe 2 only if probe 1 sustains the failure. Record OBSERVATION and UPDATED BELIEF per row; stop at the first convicted layer — deeper layers wait.
4. File the triage note: convicted layer, exonerating rows, and the contract assertion added at that boundary.

Success criterion. A layer-elimination note naming the convicted layer with its exonerating rows, plus the boundary assertion added there. A working answer without the note is explicitly not completion.

Companion tool: Model-vs-System Triage Checklist

What it accepts: the fixture + counting criterion, the frozen bundle, H1/H2/H3 with numeric FORECASTs, and the swap series in stack order with trial counts. What it performs: it enforces cheapest-first order (pipeline → params → weights → intent), checks each outcome against its FORECAST, blocks a weights conviction while any pipeline probe is untested, and stamps the convicted layer with bundle hashes. What it can establish: which layer owns this fixture’s failure under this revision — pipeline element, parameters, weights, or intent gap. What it cannot establish: line-level cause inside the convicted layer (that is Chapters 19–22 work), universal capability bounds from one fixture, or cross-revision validity. It never treats a model self-report, a single improved answer, or agreement across paraphrases as attribution evidence. How its output changes your next action: a pipeline conviction routes to Chapter 19 (inspect the assembled bytes); params routes to Chapter 21 (sampling); weights routes to Chapters 22–23 (signals, then diffs); intent routes to a spec decision before any technical fix.

Paper form, sufficient for this chapter:

FIXTURE: ___ (n=___, criterion ___)   BUNDLE: rev ___ | input hash ___ | params ___ | seed ___
H1 pipeline FORECAST: ___   H2 weights FORECAST: ___   H3 params FORECAST: ___
PIPELINE ×5: ___   PARAMS ×5: ___   WEIGHTS ×5 (same bytes): ___   INTENT spec line: ___ or UNKNOWN
CONVICTED LAYER: ___  EVIDENCE: ___  ASSERTION ADDED: ___

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. The triage discipline precedes any automation.

Reusable procedure: every “model failure” gets this triage

1. Relabel the ticket — “wrong answer” is the symptom; H1/H2/H3 are the suspects.
2. Freeze the bundle — revision, bytes, params, seed before any swap.
3. Probe pipeline first — same weights, one repaired element, ≥5 trials.
4. Probe params, then weights — same bytes throughout; weights last.
5. Check intent — quote the spec line or declare UNKNOWN before fixing.

Failure modes

• Weights-first reflex. Retraining, fine-tuning, or upgrading before the pipeline is exonerated. The most expensive fix for the cheapest defect.
• Confounded upgrade. New model + new pipeline in one deploy. The demo passes; the diagnosis is UNKNOWN and the rot compounds.
• Treating self-report as trace. “The model said it didn’t see the document, so retrieval failed.” The sentence is output, not instrumentation — verify against the assembled bytes (Chapter 19), not the confession.
• Single-failure capability ceiling. “The model can’t do X” from one case. Capability is a fixture statistic with spread, never an anecdote.
• Skipped intent. Fixing layers for behavior no spec defines. If section 4.2 is ambiguous, the fix is a policy decision with a human signature, not a prompt tweak.
• Probe-order reversal. Running the weights swap first “because the GPU team is blocked.” Costly layers convicted early exonerate nothing cheap — the pipeline probes still wait, now under schedule pressure.
• Exoneration amnesia. A layer cleared on fixture A treated as cleared for fixture B. Every new failure re-opens the triage from the top; past exonerations are filed, not inherited.

Limits, per contract: one triage convicts one layer for one fixture under one revision; it does not bound capability, does not explain the interior, and does not survive pipeline or revision changes without re-running. UNKNOWN wherever swaps moved together or trials ran single.

References

• Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12, 2024, pp. 157–173. https://doi.org/10.1162/tacl_a_00638
• Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design, or: How I Learned to Start Worrying About Prompt Formatting. International Conference on Learning Representations (ICLR), 2024. https://arxiv.org/abs/2310.11324
• Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. Seven Failure Points When Engineering a Retrieval Augmented Generation System. Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering (CAIN), 2024, pp. 194–199. https://doi.org/10.1145/3644815.3644945
• Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization. Findings of the Association for Computational Linguistics: ACL 2024, pp. 14982–14995. https://aclanthology.org/2024.findings-acl.890/
• Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are Emergent Abilities of Large Language Models a Mirage? Advances in Neural Information Processing Systems 36 (NeurIPS), 2023. https://arxiv.org/abs/2304.15004

Debugging Checklist

• Symptom relabeled: H1 pipeline / H2 weights / H3 params with distinct FORECASTs?
• Bundle frozen before any swap (revision, bytes, params, seed)?
• Pipeline probe run first, same model, one element, ≥5 trials?
• Params and weights probed only in order, same bytes held?
• No model self-report cited as attribution evidence?
• Convicted layer named with exonerating rows + boundary assertion?
• Stop rule honored (deeper probes skipped once a layer convicted)?
• Intent spec line quoted or UNKNOWN declared?

What This Chapter Established

• Attribution triage as the Chapter 4 stack applied to model systems: same-model×different-pipeline vs. same-pipeline×different-model (+params), cheapest first, weights last.
• The swap-probe discipline with pre-written numeric FORECASTs, demonstrated on the refund case convicted as pipeline (H1) — constructed illustration, no measured runs claimed.
• Lab 18 as a proposed layer-elimination record the reader executes; the Model-vs-System Triage Checklist contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: anything about the weights’ general capability, the interior mechanism, or the next revision — one fixture, one revision, one layer.
• Research grounding: context position (lost-in-the-middle, Liu et al.; mechanistically a U-shaped attention bias, partly fixable outside the weights by inference-time calibration, Hsieh et al.) and prompt format (FormatSpread, Sclar et al.) each swing measured performance enough to masquerade as a weights limit; an apparent capability cliff from one fixture can be a metric artifact (Schaeffer et al.); the pipeline suspects for probe 1 are Barnett et al.’s seven RAG failure points. A failure that moves under reordering or reformatting was never a weights failure.
• Forward link: a pipeline conviction is still vague (which pipeline element?). The next chapter opens the largest one: the actual bytes the model received.

Next

The triage convicted the pipeline — but “the pipeline” is three suspects wearing one coat: the prompt the engineer wrote, the documents retrieval returned, and the assembled context the model actually received, each transformed by templates and tokenizers along the way. Memory of what was “sent” is not evidence. The next chapter inspects the bytes.