http://programmer.ie/tags/kv-cache/ Recent content in Kv Cache on Programmer.ie: Modern AI programming Hugo en-us Wed, 23 Sep 2026 05:00:09 +0100 http://programmer.ie/books/context/09-chapter/ Wed, 23 Sep 2026 05:00:09 +0100 http://programmer.ie/books/context/09-chapter/ <p>A team running a long debugging session finally acts on Chapter 4&rsquo;s advice. Their traces show 20,000 tokens of duplicate file reads and obsolete tool output, so they add a cleanup pass that strips the duplicates before every request. Token counts fall by a sixth. The next invoice rises. Latency does not improve either. Nothing about the model changed and nothing about the task changed. The cleanup was real, the savings were arithmetic, and the bill disagrees, because the bill never measured tokens. It measured computation, and the cleanup destroyed the reuse that had been quietly subsidising every turn.</p>