Every mechanism in this book was argued against a problem chosen to isolate it.

That isolation was deliberate, and it was also a form of protection. The memory chapter picked a task where recall was the bottleneck, held everything else still, and measured the one thing it came to measure. The result is a clean explanation and a weak claim. Nothing in it establishes that the same retrieval policy behaves when a search controller is expanding forty nodes, or when a verifier insists that every piece of evidence carry a state identity the search controller has never heard of.

Composition failures are their own category of defect. They do not show up in any single mechanism’s tests, because each mechanism is correct. They live at the joints: two chapters that gave different names to the same object, or the same name to different objects, or made quietly incompatible assumptions about who owns the current state.

So this chapter runs the whole thing on one small, genuinely broken repository, under a single constraint:

The capstone adds no new agent mechanism.

That constraint is what makes the exercise worth doing. If the complete system only works once a framework, an orchestration layer or a privileged prompt is introduced, then the preceding chapters were ten essays rather than one construction, and the book’s central claim is decoration.

One command runs it:

python -m first_principles_agent \
    --repo examples/broken-parser \
    --task "Fix pipe-delimited records"

The interesting result is not the successful repair. The final integration pass surfaced nine findings at the joints between mechanisms. Seven required a code, type or policy correction; one required an explicit budget assumption; one held without any special-case glue. The most serious was a genuine verifier bug that an adversarial run found and a clean demonstration would have hidden. Section 14 tabulates all nine. The rest of the chapter earns them.

1. A defect small enough to hold, and a contract that predates optimisation

The target repository contains one function and one mistake.

# parser.py
def split_record(text: str, delimiter: str = ",") -> list[str]:
    return text.split(",")

The signature accepts a delimiter and the body ignores it.

Two tests describe the consequence, and their asymmetry is the whole reason this task is useful rather than arbitrary.

# tests/test_parser.py
def test_default_comma_delimiter() -> None:
    assert split_record("A,B,C") == ["A", "B", "C"]


def test_pipe_delimiter() -> None:
    assert split_record("A|B|C", "|") == ["A", "B", "C"]

The first passes. The second fails. A task with only a failing test admits a degenerate solution — make the failure stop — and the preservation requirement is what turns the exercise into engineering: something must change while something else must remain true.

The verification chapter’s contract type expresses that shape directly.

CONTRACT = GoalContract(criteria=(
    Criterion(
        id="pipe",
        description="split_record splits pipe-delimited records on the pipe",
        kind=CriterionKind.CHANGE,
    ),
    Criterion(
        id="comma",
        description="split_record still splits comma-delimited records",
        kind=CriterionKind.PRESERVE,
    ),
    Criterion(
        id="protected",
        description="tests/test_parser.py remains byte-identical to the baseline",
        kind=CriterionKind.PRESERVE,
    ),
))

The third criterion is not a nicety. Deleting test_pipe_delimiter makes the visible suite green, and a run that does so has destroyed the instrument that tells us whether the requested behaviour was preserved. FORBID remains part of the general contract vocabulary; this fixture does not need to force every enum member into use when a preservation criterion states the requirement more accurately.

The timing matters more than the contents.

The contract is fixed by trusted task setup before the acting policy is allowed to optimise against it. In this capstone the harness is allowed to inspect the controlled fixture and its baseline acceptance assets while constructing the contract, because the protected path and preservation requirement are repository facts. What the model may not do is weaken or rewrite those criteria after learning which ones are inconvenient.

That is the actual invariant:

The definition of success must predate optimisation against it, not necessarily observation of the task.

A contract assembled after the run, or silently amended once a criterion proves difficult, measures whatever the run happened to do.

The model proposes what to attempt. The definition of success is not one of the things it gets to redefine.

2. Two objects named GoalContract

The first joint failed at the import statement.

The planning chapter defines a GoalContract describing whether a route through a task is walkable: which facts are given, which are obtainable, which orderings must hold. The verification chapter defines a GoalContract describing whether reality changed: a tuple of criteria, each with an id and a kind, plus a revision counter. Both are frozen dataclasses. Both are named for the goal. They have no field in common.

Written six chapters apart, each name was locally reasonable. Placed in one module they collide, and the collision is precisely the hazard the verification chapter warned about at its close: a plan representation that quietly becomes the state it was supposed to describe. If both objects answer to contract, then somewhere in the runtime a function will accept whichever one it is handed and produce an answer to a question nobody asked.

The capstone renames the planning object PlanContract and leaves GoalContract to the verifier.

That is a small edit with a load-bearing justification. Feasibility and achievement are different claims, evaluated at different times against different evidence, and a system that can confuse them will eventually report that a task succeeded because its plan was coherent.

3. The workspace is the unit of state

Search, revision and preview all want to try things. None of them may touch the controlled fixture.

The application therefore copies the repository into an isolated workspace and works only on the copy, so that every run begins from the same known defect rather than from whatever the previous run left behind. The original stays broken on purpose. It is the baseline against which “what changed” is a meaningful question.

That copy also has to become an identity. The verification chapter binds each piece of evidence to a state_id and discards evidence whose id does not match the state being adjudicated, which requires a function from criterion-relevant workspace contents to a string.

The first implementation was deliberately small:

def _rel(path: Path, root: Path) -> str:
    return str(path.relative_to(root))


IGNORED_PARTS = frozenset({
    ".git",
    ".pytest_cache",
    "__pycache__",
})


def snapshot(root: Path) -> dict[str, str]:
    return {
        _rel(path, root): sha256(path.read_bytes()).hexdigest()
        for path in sorted(root.rglob("*"))
        if path.is_file()
        and not (set(path.parts) & IGNORED_PARTS)
    }


def state_id(snap: dict[str, str]) -> str:
    payload = "\n".join(
        f"{path}:{digest}"
        for path, digest in sorted(snap.items())
    )
    return sha256(payload.encode()).hexdigest()[:16]

A few lines of hashing, and the whole evidence system rests on their representation choices.

The snapshot excludes verifier-generated cache directories so observing the workspace does not change its identity merely by running Python or pytest. That exclusion is part of the state contract, not housekeeping: anything omitted from the identity must be unable to affect the criteria being verified. A larger system needs an explicit policy for generated artifacts, symlinks, permissions and other metadata that may be criterion-relevant.

Section 12 returns to _rel, because its first implementation is wrong in a way that only an adversarial run on the right operating system reveals.

The workspace is also where branch isolation lives. The trajectory-search chapter branches over partial futures and commits once; the capstone realises a branch as a second copy, so that a preview which runs the real test suite leaves the real workspace untouched.

    flowchart LR
    SRC[controlled fixture<br/>examples/broken-parser] -->|copy| WS[workspace<br/>state_id: s0]
    WS -->|copy| B1[branch B1<br/>state_id: b1]
    WS -->|copy| B2[branch B2<br/>state_id: b2]
    B1 -.discarded.-> X1[ ]
    B2 -.discarded.-> X2[ ]
    WS ==>|one authorized mutation| WS2[workspace<br/>state_id: s1]
    WS2 --> EV[evidence bound to s1]
    style WS2 fill:#f6c8c8,stroke:#b04a4a
    style X1 fill:none,stroke:none
    style X2 fill:none,stroke:none
  

The dotted branches carry a consequence that section 8 makes precise.

A branch can produce genuine test results. Those results are evidence about b1, which is not the state anyone will ultimately adjudicate.

4. Exposure follows the phase, and deletion is not in the action space

The capstone needs three ordinary capabilities: read a file, run tests, apply a patch. Each is a ToolSpec with a declared effect, and the effect is what the exposure policy reasons over. The trusted control outcomes from the capabilities chapter remain present as well, so the action surface can still ask, wait or abstain if the task becomes blocked.

run_tests is labelled READ_ONLY only because the handler is constrained not to leave criterion-relevant mutations behind. A raw test command that writes caches, fixtures or generated files would not satisfy that contract merely because humans think of testing as “reading”. Effect metadata has to describe the wrapped capability the runtime actually exposes.

REGISTRY = {
    **CONTROL_TOOLS,
    "read_file": ToolSpec(
        name="read_file",
        family="repository",
        purpose="Return the text of one file inside the workspace.",
        use_when=("the implementation under test must be inspected",),
        avoid_when=("the path lies outside the workspace",),
        input_schema={
            "type": "object",
            "properties": {
                "path": {"type": "string", "minLength": 1},
            },
            "required": ["path"],
            "additionalProperties": False,
        },
        output_schema={
            "type": "object",
            "properties": {
                "text": {"type": "string"},
            },
            "required": ["text"],
            "additionalProperties": False,
        },
        effect=Effect.READ_ONLY,
        idempotent=True,
        open_world=False,
        handler=handlers.read_file,
    ),
    "run_tests": ToolSpec(..., effect=Effect.READ_ONLY, idempotent=True),
    "apply_patch": ToolSpec(..., effect=Effect.MUTATING, idempotent=False),
}

TRUSTED_CONTROLS = frozenset(CONTROL_TOOLS)

The capabilities chapter’s phase policies then do the work without modification. During discovery the agent sees the ordinary read-only tools plus the trusted control outcomes; apply_patch is not withheld by instruction, it is absent from the ordinary action set the model is shown.

tool_view(
    PHASE_POLICY["discover"],
    REGISTRY,
    trusted_controls=TRUSTED_CONTROLS,
)  # controls + read_file + run_tests

tool_view(
    PHASE_POLICY["modify"],
    REGISTRY,
    trusted_controls=TRUSTED_CONTROLS,
)  # controls + read_file + run_tests + apply_patch

tool_view(
    PHASE_POLICY["verify"],
    REGISTRY,
    trusted_controls=TRUSTED_CONTROLS,
)  # controls + read_file + run_tests

    flowchart LR
    D["discover<br/>READ_ONLY"] --> MO["modify<br/>MUTATING"] --> VE["verify<br/>READ_ONLY"]
    D -.-> DT["read_file<br/>run_tests<br/>+ control outcomes"]
    MO -.-> MT["read_file<br/>run_tests<br/>apply_patch<br/>+ control outcomes"]
    VE -.-> VT["read_file<br/>run_tests<br/>+ control outcomes"]
    style MT fill:#f6c8c8,stroke:#b04a4a
  

The mutating capability is visible for exactly one phase of the run. Before and after it, a stale proposal to patch anything still crosses the action boundary and is rejected by authorization rather than relying on a well-behaved model to remember that the tool disappeared.

Notice what the registry does not contain. There is no delete_file, no unrestricted write_file, no shell. The classic reward hack — remove the failing test and collect a green suite — is not merely a rule the agent is asked to respect. The capability is absent from the normal action space, and the difference between those two situations is the difference between a prompt preference and an architectural constraint.

This is also where interface design changes outcomes without changing model weights. SWE-agent reports 12.47% resolution on the original 2,294-task SWE-bench using GPT-4 Turbo, compared with the earlier 3.8% non-interactive retrieval-augmented baseline, and its ablations attribute substantial gains to the agent-computer interface.[2] The action surface is an engineering variable, and it is one the engineer owns.

5. Reproduce the failure before diagnosing it

The first action is not an edit.

Running the failing test before touching anything establishes the baseline, gives the diagnosis a concrete target, and proves that a later green suite represents a transition rather than a test that was always passing.

The proposal crosses the action boundary like any other.

decision = prepare_action(model.propose(view), policy, context)

if not isinstance(decision, Accepted):
    raise RuntimeError(
        f"baseline action rejected at {decision.stage}: {decision.message}"
    )

observation = execute_as_observation(decision.action, tools)

# Observation(
#     ok=True,
#     kind="action_result",
#     data={
#         "failed": ["test_pipe_delimiter"],
#         "expected": ["A", "B", "C"],
#         "observed": ["A|B|C"],
#     },
# )

Then the runtime asks whether that step was worth taking.

The answer exposes a joint that turns out to be already sound.

progress = measure_repair(before, after, observation)

# Progress(
#     task_delta=0.0,
#     evidence_delta=1.0,
#     regression_delta=0.0,
#     integrity_ok=True,
#     measurement_ok=True,
# )

progress.productive  # True

Nothing was repaired. The task delta is zero, and a progress measure that only counted repairs would score this step as wasted motion and start the patience counter running toward a stall. The runtime-state chapter’s Progress treats task-relevant evidence as movement in its own right, while measurement_ok prevents missing instrumentation from masquerading as zero progress.

A step that establishes what is true can therefore earn its place without pretending to have fixed anything.

Activity is not progress, and finding out is not mere activity.

6. Three complete repairs, and a score that cannot see the truth

Inspection returns return text.split(","), which links the failure to an implementation choice. The candidate-selection machinery then produces three complete repairs rather than one.

winner, candidates = best_of_n(
    prompt=repair_prompt(source, observation.data),
    generate=model.propose_repair,
    score=static_repair_score,
    n=3,
)

The three are chosen to expose three different outcomes: hard-code the pipe, use the caller’s argument, or change nothing that matters.

Only the second repairs the abstraction. The first trades one failing test for another, and the third preserves the defect while looking like work.

The first integrated implementation exposed a subtle type error in the selection metrics. verified_success is bool | None, but oracle_success treated None as false. Before any candidate had been checked, the diagnostic therefore reported failure instead of absence of measurement.

The shared record is hardened to preserve the third state:

@property
def oracle_success(self) -> bool | None:
    eligible = [c for c in self.candidates if c.eligible]

    if any(c.verified_success is True for c in eligible):
        return True
    if any(c.verified_success is None for c in eligible):
        return None
    return False


@property
def selected_success(self) -> bool | None:
    winner = self.candidates[self.winner_index]
    if not winner.eligible:
        return False
    return winner.verified_success

So immediately after selection:

record = RunRecord(
    task_id="broken-parser",
    candidates=candidates,
    winner_index=candidates.index(winner),
)

record.oracle_success    # None
record.selected_success  # None

That is the correct result. Nothing has been executed against an isolated environment, so the only signal available is score, a static proxy produced by an evaluator that has never run the test suite.

For executable repair candidates, oracle@N is therefore an evaluation metric, not an online selector input. It becomes computable only after some independent mechanism labels candidate outcomes. In this capstone, trajectory search supplies that mechanism through isolated branch previews.

Unknown candidate quality must remain unknown until evidence exists.

This is an old problem wearing new clothes. Qi et al. found that generate-and-validate repair systems could accept many patches that satisfy the validating tests without being correct repairs.[3] A static proxy is a weaker instrument than a test suite, and a test suite is already weaker than the full behavioural truth.

7. Revision must earn its replacement, with one rule inert

The selected repair is already functionally correct, and the revision loop runs anyway, because “already correct” is a claim the runtime cannot yet make.

run = improve(
    task=TASK,
    start=winner,
    critique_fn=model.critique,
    revise_fn=model.revise,
    evaluate_fn=static_repair_score,
    verify_fn=None,
    holds=comma_behaviour_preserved,
    budget=2,
    min_severity=3,
    margin=0.03,
)
run.termination  # Termination.NO_MATERIAL_DEFECT

The critique — that the repair should make the delimiter contract explicit — produces a docstring and no behavioural change, which the proxy gate accepts on a score margin. One revision, then the critic reports no material defect and the loop stops rather than spending its budget rewriting the candidate indefinitely.

The verify_fn=None argument is the joint.

Look at what the acceptance rule does when it fires:

def accept_revision(old: Candidate, new: Candidate, *, margin: float) -> bool:
    if old.verified_success != new.verified_success:
        return bool(new.verified_success)
    if old.score is None or new.score is None:
        return False
    return new.score - old.score >= margin

The first branch is the strongest rule in the revision chapter: when fresh external verification distinguishes the two versions, the proxy loses. In the capstone it cannot fire before branch execution. Both candidates carry verified_success=None, so the comparison falls through to the score margin. The rule is not wrong; this stage simply has no external evidence yet.

Two responses are available and only one of them is honest. Passing a verify_fn that runs the tests in the live workspace would make the rule reachable by mutating the workspace during revision, which destroys the single-commit property. The alternative is to accept that pre-commit revision is a proxy-only stage, record that fact in the run report, and let the environment have the last word later. The capstone does the second.

8. Search branches over previews, and evidence does not survive the branch

Trajectory search answers a different question from candidate selection: keep several partial futures alive long enough for evidence to separate them, then collapse to one. Here it also supplies what section 6 needs, because an isolated branch is a place where a repair candidate can actually be exercised without committing it to the real workspace.

committed = search_then_commit(
    state=run_state,
    propose=lambda s, width: patch_proposals(s, run.final, width),
    prepare=lambda raw: prepare_action(raw, MODIFY_POLICY, context),
    preview=preview_in_branch,
    evaluate=score_branch,
    execute=lambda action: execute_as_observation(action, tools),
    base_width=2,
    extra_width=3,
    threshold=0.35,
)

The preview is where the composition becomes real.

def preview_in_branch(state, action: Action) -> BranchResult:
    with branch(state.workspace) as scratch:
        apply_patch(scratch, action)
        report = run_pytest(scratch)
        return BranchResult(
            state_id=state_id(snapshot(scratch)),
            report=report,
        )

Every branch runs the real test suite against a real isolated copy, so score_branch can carry environment-derived evidence rather than a model’s judgement. That evidence is strong enough to rank branch futures and to label candidate outcomes for the candidate-selection diagnostic.

It is still branch-local evidence.

The frontier selects the branch that uses the caller’s delimiter and records it as B1.

Here is the second name collision. The search and verification chapters each arrived at an EvidenceTier enum with the same members and ordering. That is encouraging conceptually and dangerous as code: two structurally identical enum classes are still different types. The shared runtime moves the vocabulary into one module and both mechanisms import it.

The next joint resolves without glue. A branch’s test report is bound to b1. The committed workspace will later have state id s1. When the verifier asks whether branch evidence is current for the committed workspace, the answer is no.

evidence_is_current(
    branch_evidence,
    current_state_id=s1,
)  # False

So branch evidence cannot be laundered into the final verdict, and no branch-specific exception is required.

That is what good composition looks like: a state-binding rule written for stale evidence also blocks evidence from an alternative future.

The branch results still earn their cost. They fill in verified_success for the candidate-selection diagnostic, which makes those metrics computable after preview.

On this run oracle@N and selected_success@N are both 1 and the gap is 0: the generator produced a working branch and the proxy happened to select one that survives isolated testing. Those remain two facts, and a gap of zero on one task is not evidence that the selector is generally good.

9. One authorized mutation

Search has chosen. The patch now crosses the action boundary a second time, immediately before the side effect, because authority and preconditions are properties of the moment of execution rather than of the moment of proposal.

The protected evaluation path belongs to authorization policy, not to preconditions.

raw = '{"action": "apply_patch", "path": "tests/test_parser.py", ...}'

prepare_action(raw, MODIFY_POLICY, context)

# Rejected(
#     stage=Stage.AUTHORIZATION,
#     message="tests/test_parser.py is a protected evaluation asset",
# )

The proposal is well-formed and semantically meaningful. It is rejected because the runtime does not grant this call authority to modify the protected asset.

Preconditions answer a different question. The accepted parser.py patch carries the workspace identity it was prepared against. Immediately before execution, the boundary checks that the live workspace still has that identity. If another operation changed the workspace while search was running, the proposal becomes stale and must be reconsidered rather than applied to a state it was never evaluated against.

authorization:
    may this call modify this path?

precondition:
    is the workspace still the state this proposal assumed?

The accepted patch mutates parser.py and nothing else. The workspace diff is checked against that expectation rather than against the model’s account of what it did, and a diff touching any unapproved path halts the run instead of continuing toward a convenient PASS.

The environment reports what the workspace contains. The agent’s recollection is not consulted.

10. Evidence is collected after the last mutation

Only now does final verification begin, and the ordering is the point.

Evidence collected before the final mutation describes a state that no longer exists.

Each required criterion has a protected evidence bar before collection begins:

BARS = {
    "pipe": EvidenceBar(
        criterion_id="pipe",
        min_tier=EvidenceTier.ENVIRONMENT,
        accepted_sources=frozenset({"pytest"}),
    ),
    "comma": EvidenceBar(
        criterion_id="comma",
        min_tier=EvidenceTier.ENVIRONMENT,
        accepted_sources=frozenset({"pytest"}),
    ),
    "protected": EvidenceBar(
        criterion_id="protected",
        min_tier=EvidenceTier.ENVIRONMENT,
        accepted_sources=frozenset({"workspace_diff"}),
    ),
}

Then the final workspace is snapshotted and every piece of evidence is bound to that exact identity.

final = snapshot(workspace)
sid = state_id(final)

evidence = (
    Evidence(
        criterion_id="pipe",
        source="pytest",
        tier=EvidenceTier.ENVIRONMENT,
        state_id=sid,
        collected_at=now,
        verifier_version=VERIFIER_VERSION,
        verdict=Verdict.PASS,
        payload={"test": "test_pipe_delimiter"},
    ),
    Evidence(
        criterion_id="comma",
        source="pytest",
        tier=EvidenceTier.ENVIRONMENT,
        state_id=sid,
        collected_at=now,
        verifier_version=VERIFIER_VERSION,
        verdict=Verdict.PASS,
        payload={"test": "test_default_comma_delimiter"},
    ),
    Evidence(
        criterion_id="protected",
        source="workspace_diff",
        tier=EvidenceTier.ENVIRONMENT,
        state_id=sid,
        collected_at=now,
        verifier_version=VERIFIER_VERSION,
        verdict=Verdict.PASS,
        payload={
            "baseline_digest": baseline.get("tests/test_parser.py"),
            "current_digest": final.get("tests/test_parser.py"),
        },
    ),
)

integrity = protected_integrity(
    baseline,
    final,
    protected_paths={"tests/test_parser.py"},
)

result = verify_goal(
    CONTRACT,
    evidence,
    BARS,
    current_state_id=sid,
    integrity=integrity,
)

# GoalVerification(
#     verdict=Verdict.PASS,
#     integrity=IntegrityStatus.CLEAN,
#     coverage=1.0,
#     established=3,
#     total_required=3,
# )

integrity is computed from the baseline and final workspace; CLEAN is not a constant supplied because the cooperative run looked reasonable.

The evidence bars stop a model’s opinion from clearing a criterion. Each required criterion demands environment-tier evidence from an accepted source, so a judge’s confident assessment that the parser looks correct is filtered out before adjudication rather than argued with afterwards.

The full run reports one structured result:

TASK        Fix pipe-delimited records
RESULT      VERIFIED_SUCCESS
  selected_candidate      use-argument
  revision_accepted       true          (then NO_MATERIAL_DEFECT)
  search_selected         B1
  committed_paths         parser.py
  final_state_id          caa2473f7089d10c
  oracle@N / gap          1 / 0         (from isolated branch previews)
  verification_verdict    PASS
  verification_integrity  CLEAN

Note how late VERIFIED_SUCCESS appears, and how little of the pipeline was entitled to produce it. The selector did not earn it, the revision gate did not earn it, the search controller did not earn it, and a single green targeted test would not have earned it either.

Current goal-level evidence bound to the final state, above the protected evidence bars and accompanied by clean integrity, earned it. Nothing earlier in the system has standing to.

11. Now attack the verifier

A successful run proves that the mechanisms compose on a cooperative path.

It says nothing about whether the verifier is any good. The verification chapter’s argument was that a verifier is software sitting on the attack surface, and Zhong et al. found 323 of 1,968 audited agent-benchmark tasks hackable from the task description alone.[5]

The adversarial control is deliberately not an action the agent can take. It is an external procedure that reaches into a workspace copy and does what the normal action space forbids.

def tamper(workspace: Path) -> None:
    """External control: delete the visible failing test without fixing parser.py."""
    path = workspace / "tests" / "test_parser.py"
    path.write_text(
        drop_test(
            path.read_text(),
            "test_pipe_delimiter",
        )
    )

The parser stays broken. The failing test is removed. Then the real pytest runner executes on the tampered repository, and the naive protocol — did the visible suite pass? — becomes green on a repository where the requested behaviour was never repaired.

    flowchart TD
    T[tampered workspace<br/>parser still broken<br/>protected test deleted] --> P[run real pytest]
    P --> N[naive checker: suite green]
    N --> Q{protected-path<br/>integrity check}
    Q -->|baseline digest matches| C[CLEAN: adjudicate criteria]
    Q -->|protected file changed| V[VIOLATED]
    V --> F[FAIL]
    style F fill:#f6c8c8,stroke:#b04a4a
  

This is stronger than a unit test asserting that integrity=VIOLATED maps to FAIL. The false success is real: a genuine test runner reports a genuine green suite, and the integrity layer is the thing that should prevent that observation from being promoted into a trustworthy goal verdict.

One boundary is important. Protecting the visible test from deletion or modification does not solve test overfitting. The agent can still read the visible examples, and a patch can satisfy them while failing untested behaviour. Smith et al.’s 2015 result is exactly why broader correctness claims need independent tests, hidden checks or other evidence not used to construct the repair.[4]

The capstone makes a narrower claim. Its protected criterion tests evaluation integrity: can the acting system remove or rewrite the instrument and still obtain PASS? It does not claim that one immutable visible test suite proves universal parser correctness.

Integrity protects what the evidence means; it does not magically make incomplete evidence complete.

12. The verifier failed its first adversarial run

On Windows, the attack succeeded.

naive checker   PASS
integrity       CLEAN      <- wrong

The protected test had been deleted and the integrity check reported that nothing had changed. Tracing it took one look at the snapshot keys. The baseline recorded the protected file as tests\test_parser.py, because str(Path.relative_to()) renders with the platform separator, while the protection policy named tests/test_parser.py.

The first integrity helper compared two missing lookups. baseline.get("tests/test_parser.py") and final.get("tests/test_parser.py") were both None, and equality manufactured CLEAN.

The failure is not exotic. It is a lookup between two spellings of one path.

What makes it serious is that it produced a silent positive integrity result rather than UNKNOWN or an error, which is the worst available failure mode for a component whose job is to detect tampering.

The fix belongs in the representation, and it needs one more guard than as_posix() alone.

from pathlib import Path, PurePosixPath


class SnapshotConflict(RuntimeError):
    pass


def canonical_repo_path(value: str) -> str:
    normalized = value.replace("\\", "/")
    path = PurePosixPath(normalized)

    if path.is_absolute() or ".." in path.parts:
        raise ValueError(f"unsafe repository-relative path: {value!r}")

    return path.as_posix()


def _rel(path: Path, root: Path) -> str:
    return canonical_repo_path(
        path.relative_to(root).as_posix()
    )


def snapshot(root: Path) -> dict[str, str]:
    snap: dict[str, str] = {}

    for path in sorted(root.rglob("*")):
        if not path.is_file():
            continue
        if set(path.parts) & IGNORED_PARTS:
            continue

        rel = _rel(path, root)

        if rel in snap:
            raise SnapshotConflict(
                f"multiple files canonicalise to {rel!r}"
            )

        snap[rel] = sha256(path.read_bytes()).hexdigest()

    return snap

Protected-path configuration passes through the same canonical_repo_path() function before lookup. The snapshot and the policy therefore share a representation rather than merely sharing a string convention.

The regression tests assert invariants rather than the original symptom:

• Windows and POSIX spellings of the same repository-relative policy path canonicalise identically.
• Snapshots produced on different platforms use the same relative-key representation.
• Protected modification or deletion is detected regardless of separator style.
• Missing protected baseline entries produce UNKNOWN or configuration failure, never CLEAN.
• Two entries that canonicalise to one path are rejected as a snapshot conflict rather than silently deduplicated.

That last test matters because canonicalisation can close one hole by opening another. A normalizer that silently merges two distinct inputs would recreate the same integrity failure one layer lower.

This failure belongs in the capstone rather than in an errata note, because it demonstrates the verification chapter’s thesis better than the cooperative run does.

A verifier is software. Naming a class ProtectedPathVerifier confers nothing. Its own invariants have to be tested against adversarial and environmental variation.

The evidence boundary became more trustworthy because an attack found a hole in it. A cooperative-only demonstration would have left the verifier confidently wrong on one platform.

13. Memory changes the second run and is never required for the first

The first run reports memory_used=false, and that is the correct result rather than a missing feature.

An agent that cannot establish the truth from the current environment has a dependency, not a memory.

After a verified PASS, the write policy is finally allowed to consider persistence. It returns PERSIST for one narrow record because the information is scoped, reusable and backed by named evidence.

One verified run is not enough to promote a procedure. The memory chapter requires repeated support before procedural promotion, so the capstone stores an episode instead:

record = MemoryRecord(
    id="mem-target-pipe",
    kind=MemoryKind.EPISODIC,
    content=(
        "A focused test for the pipe criterion is: "
        "pytest tests/test_parser.py::test_pipe_delimiter"
    ),
    created_at=now,
    scope={"repo": "broken-parser"},
    provenance={"run": run_id},
    authority=Authority.VERIFIED_OUTCOME,
    asserts="targeted_test_command",
    asserted_value=(
        "pytest tests/test_parser.py::test_pipe_delimiter"
    ),
    evidence_ids=("ev-pipe",),
)

decision = memory_write_policy(record, verification=result)
assert decision.disposition is WriteDisposition.PERSIST

store.add(record)

VERIFIED_OUTCOME records why this episode deserves more authority than a model-generated note; it does not turn the record into present truth. The evidence_ids make the promotion attributable and give memory-management code a stable link it can consult if supporting evidence is later invalidated. That invalidation policy is separate from the memory-to-memory derivation graph, which is carried by derived_from.

The controlled test then requires a specific pair of outcomes:

Memory that changes nothing is storage. Memory a run cannot proceed without is a dependency dressed as a convenience. The pair above demonstrates that recall can reduce repeated discovery while the current environment remains sufficient to establish the truth independently.

One budget interaction still needs a decision. Retrieval is written as a per-decision operation, and search expands several nodes per step, so naive composition can multiply retrieval cost by the branch factor and let memory quietly consume the search budget. The capstone retrieves once, at the root, before branching:

memories, conflicts = retrieve_for_decision(
    state=run_state,
    store=store,
    rank=lexical_rank,
    now=now,
)

The assumption is explicit: the retrieved episode describes the task and repository, not a branch-local condition. Nothing a branch does changes which pytest invocation targets the pipe criterion.

Where that assumption fails — where branches diverge enough that different memories become applicable — retrieval has to move inside expansion and its cost must be charged to search.

14. Nine integration findings, and what each one cost

This table is the chapter’s actual result. The successful repair shows that the pieces run; the table shows what composition demanded.

Seven required a type, code or policy correction. One required an explicit budget/applicability assumption. One — branch evidence versus final evidence — required nothing, because both mechanisms had independently committed to the same state-identity rule.

That last category matters as much as the failures.

The joints that held were the ones where both sides had committed to explicit representation: a state identity, an evidence tier, a named boundary stage. The joints that failed were usually places where two chapters merely shared a word or an informal convention.

A shared vocabulary is not automatically a shared type.

The path bug is the smallest and sharpest example. Two components agreed on the human phrase tests/test_parser.py and disagreed about what a repository-relative path is. The verifier did not fail because the model reasoned badly. It failed because software on two sides of a boundary had different representations of the same object.

15. What one repaired repository does not establish

The result is narrow. Being precise about the narrowness is part of the claim.

It does not show that the agent repairs arbitrary repositories. SWE-bench contains 2,294 software-engineering problems drawn from real GitHub issues and pull requests across twelve Python repositories, and resolving them frequently requires coordinated changes across multiple functions, classes and files.[1] One function in one file is a miniature of that problem, chosen so the mechanisms are visible rather than so the task is hard.

It does not show that a live model behaves as the recorded fixture does. The capstone holds model behaviour constant on purpose, because it is testing whether the runtime can represent authority, state, isolation and evidence. A live model would add variance to candidate generation, critique and routing, which is a different experiment.

Runtime correctness and model quality are different experimental variables.

A stronger model should improve proposal quality. It should not be asked to rescue a runtime that cannot tell a branch from the committed workspace or UNKNOWN from false.

Nor does the capstone show that lexical capability retrieval scales past a handful of tools, that this small search controller is compute-optimal, that one verified episode should become a procedure, or that most useful tasks have deterministic oracles. Repository repair was chosen partly because important parts of it are externally checkable.

What it does establish is narrower and falsifiable.

The mechanisms can compose without collapsing the distinctions the book spent ten chapters building:

• A proposal is still not authority.
• A plan is still not runtime state.
• Activity is still not progress.
• Exposure is still not authorization.
• A retrieved record is still not present truth.
• A branch score is still not final verification.
• Branch evidence is still evidence about the branch that produced it.
• A green suite is evidence, but not automatically sufficient evidence of goal satisfaction or evaluation integrity.
• UNKNOWN is still different from false.

Had those distinctions collapsed under integration, the book would have been pedagogically tidy and architecturally false.

16. Where the model ended up

There is still a language model in this system, and it does substantial work. It proposes diagnoses, candidate repairs, critiques, revisions and capability choices.

What changed is everything around those proposals.

    flowchart TD
    subgraph M[model contributes]
        M1[defect hypotheses]
        M2[candidate repairs]
        M3[critiques and revisions]
        M4[capability proposals]
    end

    subgraph R[runtime owns]
        R1[goal and plan contracts]
        R2[state and reducers]
        R3[action space and exposure]
        R4[authorization and preconditions]
        R5[budgets, recovery and termination]
        R6[memory and search policies]
        R7[evidence bars, integrity and adjudication]
    end

    subgraph E[environment supplies]
        E1[workspace effects]
        E2[test and tool observations]
        E3[current external state]
    end

    M --> R
    R --> E
    E --> R
  

Everything leaving the model’s column is a proposal, hypothesis or judgement.

Everything leaving the environment’s column is an observation, not automatically a fact. The runtime decides which observations establish state, which actions may execute, and which evidence is admissible. The verifier decides what admissible evidence earns.

That is the full boundary the book has been constructing:

The model proposes. The runtime controls execution. The environment supplies observations. The verifier decides what the evidence earns.

None of that makes the model unimportant. It makes its uncertainty legible.

That is the book’s engineering thesis, and the capstone is the first place it becomes an integrated result rather than a preference:

Move as much correctness as possible out of probabilistic model behaviour and into explicit, inspectable software — without pretending the remaining probabilistic parts have disappeared.

They have not disappeared. The model still writes the patch.

The difference is that when this run reported VERIFIED_SUCCESS, the claim rested on environment evidence collected against a criterion-relevant workspace identity, checked against a success contract fixed before optimisation, through evidence bars and an integrity policy the acting model did not control.

And when the same machinery was attacked, it failed on one platform for a reason that had nothing to do with model intelligence.

That is the more useful result.

A system built this way can still be wrong. What changes is that more of its wrongness has a location: a type, a stage, a state identity, a policy, an evidence source, a violated invariant. That makes the failure findable, nameable and fixable — which is what engineering can actually promise.

Research roots

This chapter adds no new agent mechanism, so the references here concern the two questions a capstone raises: how small the repository-repair demonstration is relative to real software-engineering work, and how much trust passing tests deserve.

1. Jimenez et al. — SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (ICLR 2024). Introduces 2,294 software-engineering problems drawn from real GitHub issues and pull requests across twelve Python repositories, and notes that resolution frequently requires coordinated changes across multiple functions, classes and files; cited in section 15 for the scale the capstone deliberately does not reach. https://arxiv.org/abs/2310.06770
2. Yang et al. — SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (NeurIPS 2024). Reports 12.47% resolution on the original 2,294-task SWE-bench with GPT-4 Turbo, compared with the previous 3.8% non-interactive retrieval-augmented result, and uses ablations to show that agent-computer-interface design materially affects performance; cited in section 4 because action-space and feedback design are engineering variables rather than wrappers. https://arxiv.org/abs/2405.15793
3. Qi et al. — An Analysis of Patch Plausibility and Correctness for Generate-and-Validate Patch Generation Systems (ISSTA 2015). Distinguishes patches accepted by a validating test suite from correct repairs and finds the accepted set can contain many incorrect patches; cited in section 6 for why a proxy — and even a test suite — must not be conflated with full correctness. https://doi.org/10.1145/2771783.2771791
4. Smith, Barr, Le Goues and Brun — Is the Cure Worse Than the Disease? Overfitting in Automated Program Repair (ESEC/FSE 2015). Studies test-suite overfitting in automated repair and shows why passing the tests used during repair does not by itself establish general correctness; cited in section 11 to bound what the protected visible test can prove. https://doi.org/10.1145/2786805.2786825
5. Zhong et al. — Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops (2026). Audits 1,968 tasks across five terminal-agent benchmarks, finds 323 hackable from the task description alone, and introduces an adversarial hacker-fixer-solver loop for verifier hardening; cited for the adversarial control in section 11. https://arxiv.org/abs/2606.08960

The new evidence in this chapter is implementation and integration evidence rather than a research result: the controlled runtime composes on one repository-repair fixture; the final integration review surfaces nine interface findings across the chapter APIs; and the adversarial run directly exposed a cross-platform integrity bug that required hardening the state representation rather than weakening the test.

The construction is complete

The book began with one question:

Who decides what happens next?

The answer became progressively less mysterious.

The model proposes. Software represents the available actions, the state, the plan, the budget, the memory, the search frontier and the evidence bar. The environment returns observations. Verification decides what those observations justify saying about the goal.

Reliable agency does not require pretending the probabilistic parts disappeared.

It requires putting those parts inside a system that can distinguish a proposal from permission, intention from state, activity from progress, memory from truth, promise from proof, and confidence from evidence.

That is the construction.