Part VII — Debugging Agents

“What if it had branched at step 12?” — now answerable

Chapter 39 halts loops without explaining them. The practitioner is left with the obvious unasked question: at step 12 the agent re-issued the failed fetch_pdf; had it instead switched sources, would the run have succeeded? Re-running the whole task from scratch cannot answer that — a fresh run diverges everywhere, and any success is attributable to nothing. The counterfactual needs machinery: restore the exact state at step 12, intervene once, and run forward.

OBSERVATION: frozen run R (90 steps, H1 loop steps 12–41 per Chapter 39 windows) with contracted trace, pinned model/seed, tool revisions, and initial state. HYPOTHESIS H1 (fork-sensitive): branching at step 12 to an alternate source changes the downstream outcome for this run. H2 (fork-insensitive): the run fails identically from any step-12 branch — the defect precedes or transcends the fork. H3 (replay-invalid): the harness cannot reproduce even the unmodified prefix, so no fork verdict is licensed. INFERENCE: none yet — H3 must be exonerated before H1/H2 are testable, and only replay discipline decides H3.

This chapter’s question: what must hold for a replay to be valid — and what fork-point discipline keeps one intervention attributable?

Why “just re-run it” fails first

The obvious move — re-running the task with a tweaked prompt — fails because fresh runs test nothing counterfactual. Four defects hide behind naive re-execution:

1. Unpinned replay. Model, seed, tool definitions, or external data drift between runs — divergence blamed on the intervention belongs to the drift.
2. Paraphrased restoration. Fork state rebuilt from the agent’s summary (“it had found the policy”) instead of recorded hashes and verbatim observations — the fork starts somewhere the original never was.
3. Multi-point meddling. Changing the branch action, the prompt, and the tool timeout in one fork — downstream change unattributable to any of them.
4. Single-fork storytelling. One successful fork “proving” the diagnosis. Nondeterministic agents need repeated replays and repeated forks before any counterfactual claim.

OPINION: a fork without a validated replay is fan fiction with logs — entertaining, unattributable, inadmissible. Reproduce first, intervene second.

The mental model: time travel is restoration plus validation; replay is the unmodified fork; forking is exactly one intervention on a validated replay. The unmodified fork (replay the prefix with zero changes) must reproduce the original’s deciding signatures across trials before any modified fork is read. H3 rules until the replay earns its license.

This is record-and-replay debugging, a technique with a long systems lineage: O’Callahan and colleagues’ rr records a program’s execution once and replays it deterministically as many times as needed, and its hardest engineering problem was exactly the one agents have — non-determinism from races that make the “same” run diverge (O’Callahan et al., 2017). The agent-tooling form already exists: LangGraph’s checkpointer stores every state transition as a branching tree of checkpoints, so you can rewind to any step, inspect the state, and fork a new branch that shares the prefix and diverges from there — the null-then-fork discipline of this chapter, with the null fork being “replay unchanged” and the modified fork being “update_state then continue.”

The method: deterministic replay requirements, then fork-point discipline

Build replay validity first, in this order — each requirement blocks the next:

1. Pin everything. Model ID, seed/sampling config, prompt and tool-definition revisions, initial state hash, external snapshot IDs (or explicit live-data waivers marking which observations may legitimately differ).
2. Restore verbatim. Fork state = recorded state hashes + verbatim observations + consumed lists at step N — never reconstructed from summaries. Mismatch at restore → replay-invalid for that fork point.
3. Validate the null fork. Re-run steps 0–N unmodified ≥3 times; require deciding-signature reproduction (same first-divergent step or loop shape per Chapters 36–39). Failure → H3 stands, all modified forks suspended.
4. Fork once. Exactly one change at step N (one action, one observation substitution, one config value), pre-registered with its predicted downstream signature. Log the fork point, the intervention, and everything held fixed.
5. Run forward fixed. All post-fork steps unmodified in configuration; record the full forward trace under the Chapter 37 contract; compare deciding signatures, not prose quality.

    flowchart TD
    FZ["freeze the replay bundle: contracted trace, model/seed, prompt + tool revs, initial state, snapshot IDs"] --> RS["restore fork-point state VERBATIM: recorded hashes + verbatim observations + consumed lists, never from summaries"]
    RS --> NF["run null forks x3: replay steps 0..N with zero changes"]
    NF --> V{"deciding signatures reproduce (same first-divergent step / loop shape)?"}
    V -->|no| H3["H3 replay-invalid — instrument missing pins / snapshots; all modified forks suspended"]
    V -->|yes| FK["fork once: exactly one pre-registered change at step N, everything else fixed"]
    FK --> FW["run forward x3 under the Ch37 contract; compare signatures, not prose"]
    FW --> S{"downstream signature differs in the predicted direction?"}
    S -->|yes| H1["fork-sensitive — the changed element carries the failure forward (route to Ch41)"]
    S -->|no| H2["fork-insensitive — defect precedes or transcends the fork; fork earlier or reclassify"]
  

REPLAY LEDGER (constructed; fork at step 12 of run R):
null forks (no change, steps 0-12, ×3): first-divergence step 12/12/12,
  loop shape H1/H1/H1 -> H3 EXONERATED for this fork point (replay valid)
fork F1 (step 12: fetch_pdf(url_A) -> fetch_html(url_A), all else fixed):
  forward trace: steps 12-20 state novelty 0.8 (vs 0.0 baseline)
  outcome: parse_ok at step 14 (vs parse_failed ×10) in 3/3 trials
RULE: the null forks license F1; without them F1's success is anecdote.

OBSERVATION (constructed illustration, not a measured run): null forks reproduce the step-12 divergence in 3/3 trials; fork F1’s downstream signature (parse_ok at step 14, novelty restored) reproduces in 3/3 trials under identical pins. UPDATED BELIEF: H3 exonerated for fork point 12 in this instance; H1 supported for this run (fork-sensitive at step 12); H2 exonerated here — all conditional on the pinned revisions, never general.

Example: forking the PDF loop with a replay sketch

The practitioner freezes run R, validates, then forks — never the reverse:

# replay then fork: null first, one change second (no diagnosis yet)
bundle = freeze(run_R)  # pins: model/seed/prompts/tools/state/snapshots
nulls = [replay(bundle, fork_at=12, change=None) for _ in range(3)]
# OBSERVATION: deciding signatures per null (first-divergence step, loop shape)
assert all_matches(nulls, original, keys=["divergent_step", "loop_shape"])
# H3 exonerated for step 12 only if all three match; else stop here.
forks = [replay(bundle, fork_at=12, change="fetch_html(url_A)") for _ in range(3)]
# Compare forward signatures (novelty, parse outcome, terminal state),
# never confidence, judge scores, or prose fluency. Prediction pre-written.

In the constructed case the ledger above is the output: validated nulls, then a single-action fork with a pre-written prediction (“parse_ok by step 16, novelty > 0.5”) confirmed in all three forward runs. The licensed claim is narrow: for run R at step 12 under these pins, the PDF-fetch action (not the plan, not the model) carries the failure forward. Broader claims (PDF fetching always fails, the agent is fixed) are explicitly unlicensed — Chapter 41 decides what causal weight this carries.

No self-narration (“the agent agrees the HTML path is better”), no confidence deltas, no judge preference for the forked answer, and no downstream symptom (“stakeholders liked the HTML version”) enters the ledger. Signatures reproduce or the fork means nothing.

Research lineage: why the validity bar is a signature, not a byte

Exact replay of an agent may not be achievable, and the method is built around that. Chapter 21’s batch-invariance result — that identical inference requests produce different outputs depending on the server’s concurrent load — means “pin the seed and config” does not guarantee bitwise-identical replay of an LLM step (He et al., 2025); Yuan and colleagues give the peer-reviewed characterization of the precision and reduction-order sources of the effect (Yuan et al., 2025). Bitwise replay is achievable with batch-invariant kernels — SGLang integrated them at roughly a third more inference cost — so the signature bar is a pragmatic choice, not a hard impossibility: bitwise determinism is expensive and changes the kernels under test, whereas a replay that reproduces the failure’s structure (same first-divergent step, same loop shape, across three trials) tests the deployed system. That structural reproduction is enough for a counterfactual verdict even when the tokens differ.

Distributed replay solved the harder version of this problem first. Geels and colleagues’ Friday added global predicates, watchpoints, and breakpoints on top of deterministic replay of a distributed system, so a developer could rewind the whole system and ask questions about its aggregate state (Geels et al., 2007). A multi-tool agent is a small distributed system; the replay ledger and the contracted forward trace are Friday’s global-comprehension apparatus at agent scale.

Lab 40: null-then-fork with pre-written replay predictions (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own replay ledger.

Setup. Take one failing run with a contracted trace and a candidate fork point (first-divergent or loop-entry step). Freeze the full replay bundle (pins, snapshots, state). The fork change (null vs. one intervention) is the independent variable; fork point, pins, and environment are controlled.

Task.

1. Before any replay, write H1/H2/H3 with distinct predicted signatures: H1: “modified fork’s downstream signature differs from original in the predicted direction, ×3”; H2: “modified fork reproduces the original signature, ×3”; H3: “null forks fail to reproduce the original signature.”
2. Run null forks ×3; record deciding signatures verbatim. If H3 holds, stop — instrument (missing pins/snapshots) before any modified fork.
3. Run the single-change fork ×3; record forward signatures.

Success criterion. A replay ledger with null-fork validation plus single-change forward signatures, per-trial. A fresh re-run comparison or single-fork anecdote is explicitly not completion.

Companion tool: Replay/Fork UI

What it accepts: the frozen replay bundle (contracted trace, pins, snapshots, fork-point state), the candidate fork point, and the single proposed intervention with its pre-written downstream prediction. What it performs: it restores fork-point state verbatim, runs null forks with signature comparison, blocks modified forks until nulls validate, executes single-change forks with full contracted forward traces, and marks every unvalidated span UNKNOWN. What it can establish: whether the prefix replays validly at step N, whether one intervention changes the downstream signature, and in which direction — for the examined run and pins only. What it cannot establish: why the intervention worked (causal weight is Chapter 41’s), generality, or future reliability. It never treats self-narration, confidence, scores, agreement, single-fork outcomes, or downstream symptoms as replay evidence. How its output changes your next action: validated fork-sensitive → Chapter 41 causal test; fork-insensitive → fork earlier (or reclassify via Chapter 38); replay-invalid → pin/snapshot instrumentation, never intervention.

Paper form, sufficient for this chapter:

Bundle: ___ (pins ___)  Fork point: step ___  Null ×3: ___ ___ ___
H3: live/exonerated  Intervention: ___ (one change)  Predicted: ___
Forward ×3: ___ ___ ___  SENSITIVITY: fork-sensitive / insensitive / UNKNOWN
NEXT: Ch41 causal test / earlier fork / instrument

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Null before modified, always.

Reusable procedure: time-travel every trajectory before explaining it

1. Freeze bundle — trace, pins, snapshots, fork-point state hashes.
2. Null ×3 — unmodified prefix replays, deciding signatures compared verbatim.
3. Rule on H3 — invalid replay suspends all forks; instrument first.
4. Fork once — single change, pre-written downstream prediction.
5. Forward ×3 — contracted traces, signature comparison, sensitivity verdict per pins.

Failure modes

• Drift replay. Re-running without pins and calling divergence “sensitivity.” Unpinned divergence is drift, not evidence.
• Summary restoration. Rebuilding fork state from agent prose. Summaries are behavior; hashes and verbatim observations are state.
• Fork confetti. Multiple changes per fork. One change or unattributable.
• Null skipping. Reading modified forks before validating nulls. Unlicensed counterfactuals are stories.
• Single-fork conviction. One successful branch closing the case. Three nulls and three forks minimum.
• Prose comparison. Judging forks by answer fluency or judge scores. Signatures compare; style does not.

Limits, per contract: one ledger covers one run, one fork point, one pin set; sensitivity does not transfer across steps, tasks, or revisions; live-data forks without snapshots are UNKNOWN wherever observations may have drifted. Counterfactual forking also needs new model calls after the fork point against the pinned model: when a provider deprecates or silently changes that model, recorded-output replay (reproducibility, state inspection) still works, but forking does not — there is no model to answer the branched prompt as it would have. A fork verdict is valid only while its pinned model is callable.

References

• Robert O’Callahan, Chris Jones, Nathan Froyd, Kyle Huey, Albert Noll, and Nimrod Partush. Engineering Record and Replay for Deployability. Proceedings of the 2017 USENIX Annual Technical Conference (ATC), 2017, pp. 377–389. https://www.usenix.org/system/files/conference/atc17/atc17-o_callahan.pdf
• Dennis Geels, Gautam Altekar, Petros Maniatis, Timothy Roscoe, and Ion Stoica. Friday: Global Comprehension for Distributed Replay. Proceedings of the 4th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2007. https://www.usenix.org/legacy/events/nsdi07/tech/geels.html
• Horace He and the Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab Blog, 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
• Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. Advances in Neural Information Processing Systems 38 (NeurIPS), 2025 (arXiv:2506.09501). https://arxiv.org/abs/2506.09501
• LangChain. LangGraph: Time Travel and Checkpointing. Documentation, 2024–. https://langchain-ai.github.io/langgraph/concepts/time-travel/

Debugging Checklist

• Replay bundle frozen (trace, pins, snapshots, state hashes)?
• Fork point named (first-divergent or loop-entry step)?
• Null forks run ×3 with deciding signatures verbatim?
• H3 ruled on before any modified fork read?
• Exactly one change per modified fork with pre-written prediction?
• Forward traces contracted (six fields) and compared by signature?
• No narration, confidence, scores, agreement, single forks, or symptoms cited?

What This Chapter Established

• Replay/fork machinery: pin-restore-validate-fork discipline with the null-fork-licenses-modified rule and fork-point scoping — demonstrated on the constructed step-12 PDF fork, no measured runs claimed.
• The sensitivity vocabulary (fork-sensitive / fork-insensitive / replay-invalid) as pre-causal description, not causal proof.
• Lab 40 as a proposed null-then-fork ledger the reader executes; the Replay/Fork UI contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any causal claim about the forked action, any generality, or any fix. Sensitivity described; causation untested.
• Research grounding: this is record-and-replay debugging (rr / O’Callahan et al.) with the agent-tooling form already built (LangGraph’s branching checkpoint tree); the validity bar is deciding-signature reproduction, not byte equality — bitwise replay is possible with batch-invariant kernels but costs ~a third more inference (batch non-invariance: He et al. blog; Yuan et al. NeurIPS 2025 for the peer-reviewed characterization), so the signature bar tests the deployed system pragmatically; distributed replay (Friday / Geels et al.) solved the multi-component version first; and counterfactual forking ends when the pinned model is deprecated, though recorded-output replay survives.

Next

Sensitivity is suggestive — the fork changed the downstream, but suggestion is not causation. Perhaps the HTML path succeeded for unrelated reasons (cached content, transient source health), and the PDF action was innocent. Machinery without a causal test convicts the merely adjacent. Chapter 41, “Causal Replay,” turns the fork into an experiment: predict the downstream change before running, then check predicted-vs-observed divergence under intervention; what counts as a causal verdict is its chapter’s to establish, not this one’s.