Part 1 — Where You Stand

The invoice nobody runs at prototype time

The paragraph review from Chapter 1 costs about 1,200 tokens in and 400 out. At prototype scale that is a rounding error, and it is why nobody computes the next four numbers.

A manuscript has 3,000 paragraphs. That is one full pass.

The review is not run once. Every edit invalidates the review of the paragraph it touched, and a careful author edits most paragraphs four or five times.

The product has users. Say a thousand, each with their own manuscript.

And it runs every month, forever, because the product is not a thing you finish — it is a thing you operate.

Nothing in that chain is exotic. Each step is a multiplication anyone could do, and almost nobody does it at the point where the architecture is still cheap to change. By the time the invoice is large enough to notice, the design that produces it has been load-bearing for a year.

So do it now, with round numbers and prices chosen only for illustration:

tokens_in, tokens_out = 1_200, 400        # one paragraph review
paragraphs, edits_per_paragraph = 3_000, 4.5
users, months = 1_000, 12
price_in, price_out = 1.00, 4.00          # $ per million tokens: illustrative, not a quote

reviews = paragraphs * edits_per_paragraph * users * months              # 162,000,000
annual = reviews * (tokens_in * price_in + tokens_out * price_out) / 1e6 # ≈ $453,600

A review that costs about a quarter of a cent becomes roughly $450,000 a year. Change the illustrative prices and the total moves proportionally. The shape does not move: the number is a product of five factors, and four of them grow with success.

What, exactly, are you buying — and does the purchase ever end?

Software used to get done

Here is the economic fact that made software the best business of the last forty years, stated plainly because we are about to break it.

Software has a high fixed cost and a near-zero marginal cost. You pay engineers for six months. Then the thing exists, and serving the ten-thousandth user costs approximately what serving the tenth did: some bandwidth and storage. The development cost is incurred once and then it is over.

That is what people mean, without quite saying it, when they say a game ships or an application is finished. Not that no further work will ever happen — that the cost of producing the thing has stopped, and everything from here is operation and improvement.

Now put a model call in the request path.

Every use of your product now costs money again. Not a fraction of a server — an actual per-use payment to a third party, priced per token, repeated forever, growing linearly with your success. The ten-thousandth user costs what the tenth did, which sounds fine until you notice that in the old model the ten-thousandth user cost nothing.

You have converted a fixed cost into a perpetual variable cost. That is not an implementation detail. It is the un-inventing of the economics that made software worth building.

This is the thing that does not add up, and it is worth being precise about where the mistake is. The mistake is not using AI. The mistake is leaving it in the finished product’s critical path, by default, without ever asking whether it needed to be there.

So the working principle for the rest of this book:

A model call in your steady-state product is a fixed cost you failed to convert.

Intelligence is a means of production

Turn that around and the strategy becomes obvious, and a bit surprising.

Intelligence used during construction is a fixed cost. It ends when the thing is built. Spend freely there — that is the best possible use of a frontier model, and it is exactly where you should be willing to pay premium rates, because the meter stops.

Intelligence used at runtime is a perpetual tax. Every call is rent on a capability you are renting rather than owning.

    flowchart LR
    subgraph BUILD["construction — fixed cost, ends"]
        B1["explore the space"] --> B2["generate candidates"]
        B2 --> B3["extract the rule"]
    end
    subgraph RUN["runtime — perpetual cost"]
        R1["only what genuinely<br/>cannot be enumerated"]
    end
    B3 -->|"baked into the artifact"| RUN
    style RUN stroke-dasharray: 4 4
  

Much of what looks like a runtime intelligence requirement is a construction task in disguise. You need to classify support tickets into eleven categories — is that a model call per ticket forever, or one afternoon with a frontier model working out what the eleven categories actually are, followed by a classifier you own? You need to extract fields from invoices — every invoice forever, or a build phase that discovers the twenty layouts covering most of your volume?

Chapter 3 already gave the test, and it is the same test: if you would be annoyed to get a different answer on a re-run, it should not be a model call. Now it has a price attached. Every operation you move from the stochastic column to the deterministic one converts a variable cost to a fixed one, permanently. The determinism default is not merely an engineering preference. It is the only cost-control mechanism in this book that compounds.

The honest limit: some products genuinely need runtime intelligence, and the criterion is precise. If the input is drawn from a space you cannot enumerate — open-ended user text or novel documents — you need the variance at runtime and you should pay for it. If the input space is narrow and you are calling a model anyway, you are renting something you could have built.

There is a second consequence, and it runs the other way. If intelligence spent on construction is a fixed cost that ends, software that was never worth building for a small audience starts to be worth building for one. The old economics amortized a high fixed cost across many users, which is why applications were built for everyone and then configured by each. Lower the construction cost far enough and a tool shaped around one person’s work can pay for itself — provided its runtime stays mostly deterministic, because that person pays the running cost too. Chapter 30 follows that thread.

What are you actually buying?

Now the question the industry mostly avoids. Models get better. Better at what, and is that the thing you need?

Start with the case that seems easiest and is not. Code has a correctness floor. It compiles or it does not. The tests pass or they do not. And once they pass, a more expensive model does not make them pass harder. There is no premium tier of “passing.”

Is there better code beyond correct? Yes — simpler and easier to change. That is real and it matters. But notice two things about it: the returns diminish quickly, and a good part of it is itself measurable in complexity and coverage. It is not an unbounded axis.

Now writing, or images, or anything whose output is judged rather than checked. There is no correctness floor at all. There is no test that passes. Every increment of capability produces something you might prefer, indefinitely, with no point at which the artifact is done in the sense that code is done. These are the domains where “the model got better” always means something.

Put those together and you get the claim I think this chapter exists to make:

Where a verifier exists, intelligence has a ceiling: the correct price is whatever clears the verifier. Where no verifier exists, intelligence has no ceiling — and your spending has no natural stopping point.

That is Chapter 4’s checkability axis, seen from the finance side. And it produces a genuinely uncomfortable corollary:

You are most tempted to buy the expensive model precisely where you are least able to measure whether it helped.

In code you can run an experiment: does the cheaper model clear the tests at an acceptable rate? Usually it does, and the decision is easy. In writing you cannot run that experiment, so you buy the best available and feel better about it — and feeling better is the entire return you have evidence for. The absence of a verifier is not only a quality problem, as Chapter 5 argued. It is a budget problem, and the two have the same root.

This is also the honest answer to “is a smarter model worth it.” It depends entirely on whether your domain has a ceiling. If it does, find the cheapest thing that reaches it. If it does not, you are in a market with no natural price discipline, and you should at least know that is where you are standing.

Floor, escalate, distill

The strategy follows, and each step has evidence behind it.

Floor. Start with the cheapest model that could plausibly work, and measure its pass rate against your verifier. Not its vibe — its pass rate. This is only possible if you built the verifier, which is why Chapter 4 put that row in bold. Commercial API prices differ by roughly two orders of magnitude, so this decision is worth more than almost any prompt engineering you will ever do.

Escalate. Send only the failures upward. Chen, Zaharia, and Zou built exactly this as a cascade with a scorer deciding when an answer is good enough to accept: their FrugalGPT could match the performance of the best individual LLM API at up to 98% lower inference cost, or alternatively improve accuracy over GPT-4 by 4% at the same cost (Chen et al., 2023). The mechanism is not clever prompting. It is refusing to pay premium rates for the many easy items in order to afford them on the few hard ones.

A cascade is only as good as the check that decides an answer is good enough, and this book eventually runs one to find out. Chapter 28’s ladder sends forty extraction items up from a deterministic rule, through free, cheap and strong models, to a person. Against sending everything to the strong model first, it was cheaper per accepted outcome under its declared cost scenario and produced one more correct acceptance. It also accepted one wrong answer its check could not recognize as wrong, and because the rule written before the run did not allow any increase in wrong acceptances, the cheaper ladder was not adopted. The two paid rungs also resolved nothing the free one had declined. Escalation saves money only where the thing deciding “good enough” is adequate to the question.

Route. Decide before the call rather than after it. Ong and colleagues trained routers on human preference data that dynamically select between a stronger and a weaker model, reducing cost by over 2× in certain cases without compromising response quality. Their routers also retained performance when the strong and weak models were swapped at test time (Ong et al., 2024).

Hold onto that last property. A router that survives having its models replaced is a router whose value does not decay when the market moves. Chapter 28 builds this book’s decision seam, and per Chapter 1’s second bet its own decision path contains no model call: a stochastic component deciding how to spend money on stochastic components is not obviously a cost control. That bet is a position, not a result, and Chapter 28 is careful to say which parts of it have been tested.

Distill. Once you know exactly what you need the expensive model to do, it stops being a dependency and becomes a teacher. Hsieh and colleagues used LLM-generated rationales as extra supervision for small task-specific models: on ANLI they outperformed the LLM baseline with a model over 700× smaller, and on e-SNLI they beat few-shot chain-of-thought with a 220M-parameter T5 — over 2,000× smaller — using 0.1% of the full dataset (Hsieh et al., 2023).

Read that as the answer to “infinitely paying for intelligence.” You do not have to. For a task narrow enough to specify, the frontier model’s contribution can be extracted once and owned, which is precisely the conversion of a variable cost back into a fixed one.

The ratchet, in order of how much it saves:

Every row down that table is more permanent than the one above it. Most teams spend all their effort on row one.

Prices fall; architecture persists

One more force, and it argues against a decision most teams make early.

Epoch AI measured how fast the price of reaching a fixed performance level has fallen, across six benchmarks. The rate varies enormously by milestone — from 9× to 900× per year. For GPT-4-level performance on PhD-level science questions (GPQA Diamond), the price fell about 40× per year (Epoch AI, 2025). Their own caveat matters: the fastest declines in that range are recent, so it is unclear whether they persist.

Take even the conservative end seriously. The capability you are paying a premium for today is likely to be substantially cheaper within a year — possibly from a different provider. Which means:

The durable engineering decision is not which model you chose. It is whether you can change your mind about it without a rewrite.

That is the entire justification for the adapter boundary built in Part 2, and it is why Chapter 12 spends a chapter on normalizing provider results into one canonical shape. A codebase that can swap models captures every price decline automatically. A codebase welded to one provider’s response format pays for the privilege of not noticing.

And the necessary counterweight, because “prices are falling” is not the same as “your bill is falling”: unit prices collapsing while usage grows is the normal history of every cheapening input. Cheaper tokens make more things worth doing, so teams do more of them. Falling prices are a reason to build for substitutability. They are not a reason to skip the arithmetic at the top of this chapter.

You cannot manage what you do not measure

The user of this book gets one model and one provider to start, and a set of costs made visible. That is deliberate, and the measurement has to be more careful than a dashboard total.

What has to be counted, per operation:

• Input and output tokens, separately — they are priced differently, often by a factor of three or more.
• Cached versus fresh input. A repeated system prompt and a novel document are not the same purchase.
• Attempts, not logical operations. One review that silently retried twice cost three times what your code thinks it did. Chapter 11 shows a real client that retries empty output without telling the caller.
• Failed calls that still bill. A response that arrived and was unusable is a cost with no output.
• Reasoning tokens, where a provider charges for tokens you never see.
• Which model, at which version, under which price, at which date.

That last one is the difference between a number and a measurement. Chapters 12 and 13 are about making these survive the trip out of three different provider response formats without being silently double-counted or zero-filled — and that chapter exists because the accounting is easy to get wrong in ways that quietly understate your spend.

Where this argument is weakest

• Cascades and routers cost engineering time. At low volume, a week of work to save $40 a month is a bad trade. The floor-escalate-distill ratchet earns its keep at volume, and volume is exactly when the architecture is hardest to change. There is no comfortable answer to that; it is a judgment call about expected scale, made early, on poor information.
• Distillation has legal and contractual limits. Provider terms frequently restrict using outputs to train competing models. This is a real constraint, it varies by provider and changes over time, and it must be checked rather than assumed.
• Distilled models are narrower. They inherit the teacher’s behavior on the distribution you distilled, and degrade off it, often without signaling it. A distilled component needs the same verifier as the thing it replaced, and needs monitoring for distribution shift.
• The strongest cost results are benchmark results. FrugalGPT’s 98% and the distillation ratios come from specific task suites. The direction is robust; your number will be different.
• “Use it during construction” assumes construction ends. For genuinely open-ended products it may not, and then the runtime cost is real and must simply be priced into the business.

Do this now

Ten minutes, one spreadsheet. Run the invoice.

For one AI feature you have built or are planning:

1. Tokens in and out for a single operation, priced separately at your provider’s current rates.
2. × operations per user per month (be honest about re-runs and edits).
3. × users at the scale you are actually aiming for, not today’s.
4. × 12.
5. Now add a column: for each operation, which row of the ratchet could remove it — trim, cheaper floor model, cascade, distill, or make it deterministic?

The annual figure is the one nobody computes while the architecture is still cheap to change. The last column is the plan.

Failure modes

• Discovering the cost curve after the architecture has set. The arithmetic is trivial and almost nobody does it while the design is still cheap to change.
• Treating the frontier model as the default. It is the correct choice for construction and the expensive choice for steady state.
• Buying capability in a domain with no ceiling and calling it an investment. Without a verifier there is no evidence the upgrade helped, only a feeling that it did.
• Optimizing prompts while ignoring model choice. Prompt trimming adjusts a coefficient; provider prices span two orders of magnitude.
• Measuring logical operations instead of attempts. Hidden retries make your cost model wrong in the direction of optimism.
• Welding to one provider’s response shape. Guarantees you pay full price through every price decline.
• A router that calls a model to decide. You have added cost to the decision about cost.

What this chapter established

• Software’s economics were high fixed cost and near-zero marginal cost. A model in the request path converts a fixed cost into a perpetual variable one — the un-inventing of why software was worth building.
• A model call in your steady-state product is a fixed cost you failed to convert. Intelligence is best treated as a means of production: spend freely during construction, where the meter stops; sparingly at runtime, where it never does.
• The runtime test is whether the input space is enumerable. Unenumerable input genuinely needs runtime variance; narrow input is rent on something you could have built.
• Where a verifier exists, intelligence has a ceiling and you should pay only what clears it. Where none exists, spending has no natural stopping point — and that is exactly where the upgrade is most tempting and least measurable.
• The ratchet: trim, floor, cascade or route, distill, and finally make it deterministic. Each step is more permanent than the last. Cascades reached up to 98% cost reduction at matched performance; routing over 2× at unchanged quality; distillation produced models 700–2,000× smaller that beat their teachers on the distilled task.
• Prices for fixed capability have fallen 9×–900× per year depending on the milestone, so substitutability is worth more than model selection — while cheaper units do not imply a smaller bill.
• You cannot manage cost you do not measure at the attempt level, including cached input, hidden retries, billed failures, and unseen reasoning tokens.

Next

One thing is still missing, and without it everything in this chapter is unusable.

You cannot pay only what clears the verifier if you have not built the verifier. You cannot tell whether the cheaper model was good enough, whether the distilled model held up, or whether this month’s process is better than last month’s. Vendors sell capability, which is a scalar. Aiming it requires a measurement, and measuring a stochastic process is harder than measuring a deterministic one — hard enough that most teams quietly skip it and buy capability instead.

The last chapter of Part 1 is about how you would know.

Continue with Intelligence in the Wrong Direction.

References

• Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176, 2023. https://arxiv.org/abs/2305.05176
• Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665, 2024 (ICLR 2025). https://arxiv.org/abs/2406.18665
• Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Findings of the Association for Computational Linguistics: ACL 2023, pp. 8003–8017. https://aclanthology.org/2023.findings-acl.507/
• Epoch AI. LLM Inference Prices Have Fallen Rapidly but Unequally Across Tasks. Data Insight, March 12, 2025. https://epoch.ai/data-insights/llm-inference-price-trends