Part 1 — Where You Stand

An uncomfortable piece of bookkeeping

Software engineering was the first knowledge profession to be substantially automated by language models. The usual explanation is that code is logical and therefore tractable for a machine.

That explanation is wrong, and believing it will cost you, because it points at the wrong property when you go looking for the next domain.

Here is what actually happened. For roughly two decades the industry adopted a family of methodologies — Agile, Scrum, Kanban, whatever your organization called its version — that mandated a particular discipline of record-keeping. Work had to be:

• broken into units small enough to finish in days,
• each given a natural-language description of what was wanted,
• each given acceptance criteria and a definition of done,
• tracked through an explicit state machine,
• discussed in writing, in comments and reviews,
• and closed by a specific, linked change to the codebase.

Developers experienced this as project management. Some of them resented it. Read the list again as a data schema:

The ceremony	What it actually produced
Ticket description	A natural-language specification
Acceptance criteria / definition of done	A verifier
The merged diff that closed it	A solution
Comments, reviews, state transitions	A trace of how it was reached

For twenty years the software industry produced (specification, verifier, solution) triples, at enormous volume, in public, and filed them under administrative overhead.

Nobody did this to train a model. That is the point. The corpus was a byproduct.

Which domains are ready to be automated, and how would you tell before you spent the money finding out?

The receipt

This is not an argument by analogy. You can look at how the standard benchmark for AI software engineering was built.

Jimenez and colleagues constructed SWE-bench by taking roughly 90,000 pull requests from 12 widely-used Python repositories and filtering them on four criteria: the PR resolves a GitHub issue, it contributes tests, the project installs, and the tests exhibit a fail-to-pass transition — failing before the change and passing after it. What survived was 2,294 task instances, each evaluated with FAIL_TO_PASS and PASS_TO_PASS tests (Jimenez et al., 2024).

Look at what each filter is doing. Resolves an issue selects for a written specification. Contributes tests selects for a written verifier. Fail-to-pass confirms the verifier actually discriminates between the before and after states. The diff is the reference solution.

Nobody wrote those 2,294 specifications for a benchmark. Developers wrote them to close tickets. The benchmark is a harvest of work that had already been done, in a format that existed for entirely unrelated reasons.

That is the shape of the thing. When a domain has been keeping this kind of book on itself, an enormous amount of the work of automating it is already finished before anyone starts.

Why code fell first

So: not because code is logical. Plenty of things are logical.

Code fell first because it has the cheapest verifiers in existence.

Consider what a software engineer has that almost no other professional has. A compiler that rejects a large class of wrong answers for free, in seconds, before anything runs. A type system that acts as a proof checker maintained by the language team. A test suite that is a specification you can execute. Continuous integration that runs all of it on every change, unattended, with no human in the loop. Version control that makes a wrong attempt cheap: branch it, run it, revert to green in seconds.

Chapter 3 gave the productive architecture: where you cannot write the solution but can write the check, you put a stochastic generator behind a deterministic verifier. Software is the domain where humans had already built the verifier — decades early, for their own reasons, at great expense, and with no sense of what it would later be used for.

This reframes the whole question. The axis is not difficulty. It is checkability.

That distinction has teeth. Competitive programming problems are far harder than most production tickets, and they fell earlier, because they ship with test cases. Meanwhile a task as cognitively trivial as “file this expense to the right cost center” resists automation in many organizations — not because it is hard, but because nobody can cheaply tell whether it was done right.

How much of the work the verifier does

It is worth knowing the size of the effect, because it determines where you should spend your engineering effort.

AlphaCode attacked competitive programming by generating vast pools of candidate programs, then filtering them against the example tests included in the problem statement. Filtering on those example tests removed more than 99% of the model’s samples. With 10⁶ samples per problem, the largest 41B model produced at least one solution passing the example tests for over 90% of problems. Those survivors were clustered down to at most 10 submissions, which achieved an average ranking in the top 54.3% of human participants on Codeforces (Li et al., 2022).

Read that as an accounting statement rather than a headline.

The generator’s raw output was under 1% acceptable. Essentially all of the usable correctness in the final result was contributed by a deterministic filter that cost nothing to run and contained no intelligence whatsoever. The model supplied variance; the tests supplied correctness.

Bound it: competitive programming, self-contained problems, example tests handed to you in the problem statement, and a compute budget no ordinary project will spend. A million samples is not a strategy anyone should copy. The ratio is the lesson, not the method.

And the lesson is the most practical thing in this chapter. If you are deciding where to invest in a new domain, and the domain’s verifier is weak, then buying a better model is buying more of the component that was not producing the correctness. Later chapters measure pieces of this directly: Part 5 finds more model variety adding little or nothing on its first corpus, and Chapter 28 finds a carefully built check accepting a wrong answer that no amount of extra model capability would have flagged.

Recent benchmarks make the same asymmetry explicit. BrowseComp is built from 1,266 questions whose answers are hard to find but short and easy to check; its question writers had to confirm that GPT-4o, with and without browsing, OpenAI o1 and an early deep-research model could not solve them (OpenAI, 2025). Trainers working on questions they had not written solved 29.2% within a two-hour limit. GPT-4o scored 0.6%, GPT-4o with browsing 1.9%, o1 9.9%, and the deep-research agent 51.5%. It is a vendor benchmark of short answers, not a workplace claim, but it states this chapter’s asymmetry cleanly: generation expensive, checking cheap — provided someone manufactured the check. PaperBench shows what manufacturing the check costs: replicating 20 ICML 2024 papers is graded against rubrics broken into 8,316 individually gradable tasks, co-developed with each paper’s authors, and the automated judge is validated on a benchmark of its own; the best agent averaged 21.0% (Starace et al., 2025). SWE-Bench Pro pushes coding evaluation toward longer work: 1,865 human-verified problems from 41 repositories, some expected to take a professional hours or days (Deng et al., 2025). The through-line is that each of these required enormous effort spent building the verifier, before any generator could be measured against it.

Reading terrain

Now the general question, which is what the rest of this book is for.

Anywhere a deduction has to be made over text or recorded information, there is an opening — writing, research, adjudication, review, fraud detection. That instinct is right, and it is the first thing to look for.

But text and deduction alone get you a demo. Text, deduction, and a cheap check get you a process. The difference is the entire subject of this book, and it is why the terrain survey has five questions rather than one.

Survey question	Software’s answer	If the answer is no
Is the work already expressed in text or symbols?	Tickets, code, docs, reviews	The steamroller has no purchase. Your first job is codification, not model calls.
Does it decompose into independently attemptable units?	One ticket, one diff	You cannot localize a failure or retry cheaply. Everything is one big bet.
Is there a verifier cheaper than doing the work?	Compiler, types, tests, CI	You have a demo, not a process (Chapter 21).
Is a wrong proposal cheap to discard?	Branch it, revert it	Every attempt is a production incident. You need authority gates before you need intelligence (Chapter 20).
Is there volume, with variation?	High volume with variation	Low volume: not worth a process. No variation: write a script (Chapter 3).

The middle row is load-bearing. It is the one people skip, and skipping it is the single most reliable way to build something that demos beautifully and cannot be deployed.

Apply the survey honestly to the openings listed above and they do not all score the same:

• Research and literature synthesis. Text, yes. Decomposes per claim, yes. Verifier: better than people assume — a cited source either contains the claim or it does not, and checking that is mechanical. Strong terrain, and it is exactly the running example this book builds.
• Adjudication, claims handling, compliance review. Text, yes. Decomposes per case, yes. Verifier: the cited rule either applies to the recorded facts or it does not, and appeal outcomes provide delayed ground truth. Strong terrain — but authority-heavy, because a wrong proposal is not cheap to discard when it has already been sent to a claimant. That is Chapter 20’s problem.
• Evaluation and grading. Text, yes. Verifier: a rubric, plus inter-rater agreement. Workable, with a trap — the “verifier” here is often just another judgment, and Chapter 21 argues at length that agreement among generators is not evidence.
• Fraud detection. Text, yes. Volume, enormous. But the verifier is delayed and adversarial: you learn you were wrong months later, if ever, and the terrain actively changes shape to defeat whatever you deployed. This is the case that looks perfect on a naive reading — huge volume, text-shaped, deduction required — and is genuinely hard for reasons the first four rows do not reveal.

Keep that last one. A survey that returns “yes” for everything is not a survey.

You are not holding a chatbot

Here is the turn, and it is the reason this book exists.

Almost everyone is looking at the chat window and asking whether the thing behind it is smart. Is it conscious, is it reasoning, did it really understand the question, will it pass some exam. That is an interesting argument and it is the wrong object.

The chat window is the demonstration interface. It is the debug console that happened to ship to consumers first, because it was the fastest way to let a person see that the thing worked at all.

What you actually have is a machine for flattening codified terrain — and a world absolutely full of uneven, badly-organized, text-shaped information that nobody has rolled yet. Software’s terrain was unusually tidy, so it went first. Most domains have the text and lack the bookkeeping.

Which produces the conclusion that took me longest to accept, and that most reorders what you do on Monday morning:

In a new domain, the first job is usually not to call a model. It is to build what Scrum built.

Units. Descriptions of what is wanted. Acceptance criteria. A record of what was attempted and what happened. A way to tell finished from unfinished. You produce the terrain before you roll it.

And that is precisely what the rest of this book constructs. The context package that names what a call may see (Chapter 15). The ledger that records what happened (Chapter 16). The artifact store that preserves the observation (Chapter 17). Claims with evidence levels, so a suggestion is distinguishable from a checked fact (Chapter 18). Verification bound to a target state (Chapter 21). Read as a whole, the runtime in this book is a machine for manufacturing the bookkeeping that makes a domain rollable — for domains that never had a Scrum.

The road consumes itself

One honest complication, and it is the most interesting finding in the chapter.

Del Rio-Chanona, Laurentsyeva, and Wachs measured what happened to Stack Overflow after ChatGPT’s release, using a difference-in-differences design against Russian and Chinese counterpart sites where ChatGPT access is limited, and against mathematics forums where the model was less capable. They estimate a 16% decrease in weekly posts on Stack Overflow, an effect that increases in magnitude over time and is larger for the most widely used programming languages. Posts made after ChatGPT receive similar voting scores to those before, so this is not merely the displacement of duplicate or low-quality content (del Rio-Chanona, Laurentsyeva & Wachs, 2023).

Sit with the shape of that. The decline is largest exactly where the model was strongest, which is exactly where the public corpus was richest. The steamroller flattens hardest where the terrain was most thoroughly codified — and in flattening it, stops it being replenished. The authors’ own framing is that knowledge is shifting from public repositories into private interactions.

Bound it: one platform, a specific window after release, an estimate from a model with chosen controls rather than a randomized experiment, and the counterfactual platforms differ from Stack Overflow in more ways than ChatGPT access.

The consequence for you is direct and it is not abstract. If the public codified layer stops growing where it was thickest, the records your own process keeps become more valuable, not less. A ledger of what your organization tried, what was checked, and what turned out to be true is the private replacement for a commons that is being drawn down. That is a considerably better reason to build the record-keeping of Parts 2 and 3 than “good engineering practice.”

Do this now

Twenty minutes. Run the terrain survey on your own domain.

Take the work you actually want to automate — not a toy — and answer the five questions honestly in writing:

1. Is it already expressed in text or symbols?
2. Does it decompose into independently attemptable units?
3. Is there a verifier cheaper than doing the work? Name it. If you cannot name it, write “none”.
4. Is a wrong proposal cheap to discard?
5. Is there volume, with variation?

A worked yield calculation grounds the survey, using only numbers already cited above: SWE-bench kept 2,294 instances from roughly 90,000 pull requests, so 2294 / 90000 ≈ 0.0255, about 2.5% harvested and about 97.5% discarded by the four filters. That percentage is not a target; it is what a strict verifier costs when applied honestly.

Apply the same arithmetic to your own shelf before spending on models. Pull your last 40 work items and count how many have all three — a written request, a check cheaper than the work, and a recorded outcome:

Count	Meaning
Triples / 40	Your harvest yield at current bookkeeping
0–2 triples	Terrain missing: first increment is bookkeeping, not prompts
3+ triples	A pilot has something to filter against

The reader can now do one new thing: price the verifier before pricing the model, and choose between designs — build bookkeeping first, or admit the result will be a demo.

If row 3 says “none”, stop and answer a different question first: what would a verifier for this look like, and what would it cost to build? That answer, not a prompt, is your first increment — and Chapter 7 turns it into something you can run.

Failure modes

• Choosing a domain by difficulty instead of checkability. Hard-but-checkable beats easy-but-uncheckable.
• Skipping the verifier row of the survey. Produces a system that demos perfectly and can never be trusted in production.
• Buying a better model to fix a weak verifier. AlphaCode’s filter removed >99% of samples. More samples through a bad filter is more cost for the same correctness.
• Treating another model’s agreement as the check. Covered properly in Chapter 21; noted here because it is the standard way the verifier row gets faked.
• Assuming the corpus that made this domain tractable will still be there. It is measurably shrinking where it was richest.
• Calling a model before the terrain exists. In an uncodified domain, the first increment is units, descriptions, and acceptance criteria — not a prompt.

What this chapter established

• Software automated itself first because of bookkeeping, not difficulty. Two decades of Agile ceremony produced (specification, verifier, solution) triples at industrial scale, as a byproduct.
• The receipt is visible in SWE-bench’s construction: ~90,000 PRs filtered on issue resolution, test contribution, installability, and a fail-to-pass transition, yielding 2,294 instances.
• The operative property is checkability, not logicality. Code has the cheapest verifiers ever built: compilers, types, tests, CI, and cheap rollback.
• The verifier does most of the work. AlphaCode’s example-test filter removed over 99% of samples; the model supplied variance, the tests supplied correctness.
• Terrain survey: codified, decomposable, cheaply checkable, cheap to be wrong, high-volume-with-variation. Fraud detection passes four rows and fails on a delayed adversarial verifier.
• You are not holding a chatbot. You are holding a machine for flattening codified terrain, and in most domains the first job is to manufacture the terrain.
• The public corpus is contracting fastest where it was richest, which makes your own recorded evidence more valuable than it used to be.

Next

We know which terrain is worth rolling, and we know the architecture: a confined stochastic proposal step behind a deterministic check. In most domains, though, the deterministic check bottoms out somewhere in a person — no compiler exists for “is this claim adequately supported.”

That person is the last verifier in the chain, and the last verifier degrades. Not through laziness, and not in a way that discipline fixes. The next chapter is about the specific way it happens, why success causes it rather than failure, and what has to be true of your process for review to survive contact with a system that is usually right.

Continue with Meat Proxy.

References

• Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? International Conference on Learning Representations (ICLR), 2024. https://arxiv.org/abs/2310.06770
• Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, et al. Competition-Level Code Generation with AlphaCode. Science, vol. 378, no. 6624 (2022), pp. 1092–1097. https://doi.org/10.1126/science.abq1158
• OpenAI. BrowseComp: A Benchmark for Browsing Agents. arXiv:2504.12516, April 2025. https://arxiv.org/abs/2504.12516
• Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s Ability to Replicate AI Research. arXiv:2504.01848, 2025. https://arxiv.org/abs/2504.01848
• Xiang Deng et al. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv:2509.16941, 2025. https://arxiv.org/abs/2509.16941 — 1,865 problems, 41 repositories.
• Maria del Rio-Chanona, Nadzeya Laurentsyeva, and Johannes Wachs. Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow. arXiv:2307.07367, 2023. https://arxiv.org/abs/2307.07367