The Context Window Is a Budget

Wed, 23 Sep 2026 05:00:04 +0100

Chapter 3 ended with an observation: every hidden layer costs something. A team running a coding agent on a model with a million-token window read that sentence and shrugged. Their sessions rarely passed 150,000 tokens. Nothing was being truncated, nothing was overflowing, and the window, they reasoned, had settled the matter. Then two things happened. First, a long debugging session died at turn eleven with the model stopping mid-repair, not because the input had crossed a million tokens but because the input plus the output the model still needed no longer fit together. Second, the invoice arrived, and the “plenty of headroom” sessions turned out to be the most expensive ones they had ever run. The window had not settled anything. It had merely set the outer wall of a much smaller room they were actually living in.

Allocation on Programmer.ie: Modern AI programming

The Context Window Is a Budget