Prefix Caching

With a 512-token prefix already in the cache, a 32-token follow-up got its first token in 22 milliseconds. Without it, the same request took 37. A second tenant’s request, probing the same prefix, could tell the difference every single time: in 40 trials, the probe that hit a cached prefix was faster than every one that did not.

The cache is a speed-up, and the speed-up is a measurement anyone can make. If two tenants share a prefix cache, either can find out, by timing, whether the other has recently sent a particular prompt. Giving each tenant its own scope removes that signal. In the same experiment the scoped probe was indistinguishable from a cold one (the chance that a cold probe was slower than a scoped one was 55 per cent, close to a coin toss), and it costs a share of the hit rate that depends heavily on how big the cache is.

This chapter builds a real prefix cache on Chapter 14’s blocks, checks that reusing a prefix changes nothing the model computes, measures what it saves, and measures both what isolation costs and what the absence of it gives away.

Where this sits

Chapter 10 established that a cache is only valid for the exact computation that produced it, and defined prefix_key(manifest, tokens) as its identity. Chapter 14 built blocks with reference counts and copy-on-write, and forked one prefix into two branches. Chapter 15’s scheduler runs many requests, and many of them in an agent or chat setting begin with the same system prompt and the same earlier turns. Chapter 16 asks what it is correct to share between requests that were not forked from each other, how long to keep it, and with whom.

The Inference Lab piece added here is the prefix index: a map from chained block hashes to physical blocks that holds one reference per cached block, adopts cached blocks into a new sequence without copying, registers new blocks as prefill completes them, and evicts the least recently used blocks that nothing is using. Beside it sit a token-level model of the same index for studying hit rates, and the timing experiments.

For the Pi series, this is the chapter where an agent’s habits become a cost. An agent that sends the same thousand tokens of instructions and tool definitions with every call pays for them in prefill every time unless the engine remembers. And one that runs for several users is sending those users’ prompts through a shared component.

What can be shared

The keys and values at a position depend on every token up to and including it, and on nothing after. Chapter 4 showed this is exact (changing the last token left every earlier logit identical, to zero) and Chapter 10 used it to justify the cache. It follows that two requests whose prompts begin with the same tokens have the same keys and values for those positions, provided that everything else that determines them is also the same. That condition is where the risk is.

What determines a cached key or value:

• The tokens of the prefix. Every token, in order.
• The positions. GPT-2 adds a learned position embedding, so the same tokens at different offsets give different states. In models with rotary positions it is the same: the position is built into the key.
• The model artefact. Weights, as Chapter 2’s manifest records them; a changed layer, a different dtype or a different quantisation scheme (Chapter 12) gives a different state.
• Anything that transforms the prompt. The tokenizer, a chat template, an adapter, image embeddings for a multimodal prompt.
• The tenant, if different trust domains must not learn from each other’s cache.

I call the combination of everything but the tokens the scope. A cache cannot know what belongs in it: the scope is a string composed by the caller, and the cache will happily share between any two requests whose strings match. The practical content of this chapter is that the string must be assembled with the same care as the artefact manifest.

A key that includes its history

The block-level key is a chained hash:

so a block is found only when the scope, every earlier block and its own tokens are all identical. The chaining carries the position and the history at once. Keying a block by its own tokens alone is the tempting mistake, and it is wrong in a way that a test shows directly: two prompts that differ in their first block and agree in their second have the *same* unchained key for the second block, and a different chained one. Sharing on the unchained key would hand a request the keys and values computed under a different history, which is the "cache built for another prompt" failure of Chapter 10, with output errors of the order of ten logits and the right next token in 40 per cent of trials.

vLLM’s automatic prefix caching, which its design documentation describes, uses the same construction: each full block is hashed from its parent block’s hash, its own tokens, and any extra values that make it unique, such as adapter identifiers, multimodal input hashes and a cache salt; only full blocks are cached; the default hash is SHA-256; eviction is least-recently-used. Mine differs in the details (the scope enters every block’s hash, where vLLM injects the salt into the first block’s and lets the chain carry it) but not in the idea.

The mechanism

Three operations on the Chapter 14 pool.

Lookup. Hash the prompt’s full blocks in order and follow the chain through the index until the first miss. The result is a list of physical blocks covering the longest cached prefix. It never covers the final prompt token: the model must compute at least one position to produce the logits for the next token, so a prompt of exactly four blocks can reuse at most three.

Adopt. The new sequence takes the matched blocks into its block table by reference. No data moves and no arithmetic happens for those positions; each block’s reference count rises by one.

Register. After prefill, each newly completed full block is added to the index, which holds its own reference. The block therefore survives the sequence that created it. A trailing partial block is not registered, since it is still being written.

Evict. When the pool is short of free blocks, the index drops the least recently used blocks whose only reference is its own. A block that a running sequence uses has a higher count and is never evicted. After all sequences finish and the index is emptied, the pool must be exactly as empty as it began, and a test checks that.

It changes nothing

Eight requests shared a 300-token system prompt followed by 20 to 69 tokens of their own. The first request found nothing and computed all 320 positions. Each of the other seven reused 288 tokens (18 full blocks of 16; the system prompt’s last 12 tokens fall in a partial block and are recomputed). Compared with a full prefill, the largest logit error was \(7.6 \times 10^{-5}\) and the argmax matched every time. That is the same rounding-level disagreement as every other cache comparison in the book. The unit tests add the others: another scope never hits, shared blocks carry one reference per user plus the index’s, and a block in use is never evicted.

What it saves

Time to first token for a request with a prefix of \(P\) tokens and a 32-token suffix of its own, when an earlier request has already cached the prefix (a hit) and when not (a miss). Median of 15 trials per cell, on an empty pool each time:

Cached prefix	Miss	Hit	Speed-up
128 tokens	22.1 ms	22.1 ms	1.00
256 tokens	25.4 ms	22.3 ms	1.14
512 tokens	37.1 ms	22.0 ms	1.68
768 tokens	57.3 ms	22.2 ms	2.58

A hit costs about 22 milliseconds whatever the prefix length, which is the floor of Chapters 7 and 9: a launch-bound forward pass over the 32 suffix tokens. A miss pays that floor plus the prefill of the prefix, which is itself flat up to about 256 tokens and linear above. So on this machine, caching a prefix of 128 tokens saves nothing, and caching one of 768 saves 35 milliseconds. The benefit starts where the saved prefill exceeds the floor.

That is a result for a model small enough to be launch-bound. For a larger model, where prefill is compute-bound from the first token, the saving grows with the prefix from the start, and the arithmetic is easy: the saved time is the prefill time of the cached tokens. What does not depend on the model is the ranking of what to cache: the longest shared prefixes first, since the fixed cost of a request is paid either way.

How often it hits

Whether it is worth having depends on how often requests share a prefix, and whether the cache is big enough to still hold it when the next one arrives. I modelled this with the same index logic and no tensors, on a synthetic agent-style workload:

• 120 sessions spread over 5 tenants; each session picks one of 8 system prompts of 512 tokens, with popularity proportional to \(1/k\) (a Zipf distribution), and has 1 to 6 turns;
• each turn’s prompt is the system prompt, every previous turn, and 90 new tokens; the reply (90 tokens) joins the history;
• sessions interleave at random, giving 437 requests with a mean prompt of 906 tokens.

The token hit rate is the fraction of prompt tokens served from the cache. With the 8 system prompts shared by all tenants:

Cache size	One shared scope	One scope per tenant
1,024 tokens	8.0%	2.7%
4,096 tokens	38.2%	15.1%
16,384 tokens	66.4%	46.5%
65,536 tokens	81.1%	77.9%

Three things are visible.

Size matters more than anything else. A cache of 1,024 tokens holds two system prompts’ worth and thrashes. By 65,536 tokens, which for GPT-2 in fp32 is 4.5 GiB of cache at 72 KiB per token, the hit rate saturates.

Isolation has a price that shrinks as the cache grows. At 4,096 tokens, a scope per tenant cuts the hit rate from 38 to 15 per cent, because each of the five tenants stores its own copy of the same system prompts, and the cache is five times as crowded. By 65,536 tokens the cost is three points. A deployment that needs isolation either pays for a bigger cache or accepts a lower hit rate; it does not get both for free.

When there is nothing to share, scope costs nothing. If each tenant has its own system prompts, the two columns are identical to the digit (2.7, 15.1, 46.5, 77.9 per cent), because the shared-scope cache had no cross-tenant hits to lose.

The ceiling of 81 per cent is partly an artefact: in this model only prompt blocks are registered, so the keys and values of the generated replies are not in the cache, and the next turn must recompute them. A real engine could register decoded blocks too. I have understated the reuse across turns by an amount that I have not measured.

What it gives away

A hit is faster than a miss. That is the point of the cache, and it is a signal.

Imagine an attacker who can send requests to a service whose prefix cache is shared across customers. To test whether another customer recently used a particular 512-token prefix, the attacker sends it, adds a little text of their own, and times the first token. If the prefix was cached, the answer comes in the hit time; if not, in the miss time. I ran that experiment with 40 trials per condition on a quiet machine:

Condition	Median first-token time	Range
Victim used the prefix, attacker in the same scope	22.6 ms	21.6 – 27.9 ms
Victim used the prefix, attacker has their own scope	36.2 ms	35.5 – 37.7 ms
Nobody used the prefix	36.4 ms	35.4 – 40.5 ms

With a shared scope the two distributions do not overlap: the slowest hit (27.9 ms) was faster than the fastest miss (35.4 ms), and the probability that an uncached probe is slower than a hit was 1.00. An attacker with a stopwatch learns the answer with certainty. With the attacker in a different scope, the timing is the same as nobody having used the prefix, and the probability falls to 0.55, which is indistinguishable from guessing.

Three qualifications.

This is the best case for the attacker. One process, no other traffic, a 512-token probe, and a hit that saves 14 milliseconds of a 37 millisecond request on a launch-bound model. In a loaded production system, queueing noise would blur a 14 ms difference, and the attacker would need more probes. It would not make the signal go away, and a larger model saves more.

The scope is what prevents it, and nothing checks it. The cache treats the scope as an opaque string. The salt-per-request option in vLLM’s design plays the same role as a per-tenant scope here: according to its documentation it isolates tenants and helps prevent timing-based inference of cached content. It mitigates the leak between tenants. It does not replace authorisation, and it does not stop a tenant from observing its own cache; a malicious user within a tenant is not excluded.

Other channels remain. I measured one. A tenant can also fill the cache to evict others’ entries and slow them down, and cached contents are derived from private prompts, which makes a memory dump or a deletion request subject to the same questions as any stored user data. None of this is implemented, measured or solved here.

Provider caching

A hosted API shows the same idea at a different layer. Anthropic’s prompt-caching documentation describes, for the caller, breakpoints placed with cache_control, a default lifetime of five minutes that each use refreshes, with a one-hour option, a requirement that the cached prefix be identical in text and images in the order tools, system and messages, minimum cacheable lengths that vary by model, and usage fields that report tokens written to and read from the cache. It also states that caches are not shared between organisations, and that within an organisation they are isolated by workspace on some platforms and not others.

That is a description of an interface and its semantics, and it should be read as one. It tells the caller what must match for a hit, how long it will last, how to see the effect in the usage numbers, and what scope the provider isolates at. It does not say how the provider stores the state, and I do not infer it. The caller’s control is over breakpoints and lifetimes, not over physical blocks.

What the numbers do not show

• A small model, launch-bound. The saving per hit is bounded by the floor, and larger models would save more.
• One workload, synthetic. Zipf-popular system prompts, fixed turn lengths, five tenants, and registration of prompt blocks only. Real traces differ in every one of those.
• One timing attack, best case. No concurrent load, one prefix length, forty trials.
• The scope is only as good as its author. Nothing here checks that a scope contains the artefact manifest, adapter or tokenizer.
• No poisoning, denial-of-service, deletion or persistence analysis.
• No multimodal prefixes, and no check of how image embeddings would enter the key.
• LRU only. vLLM’s documented refinement, evicting the last block of a finished request first because it is least likely to be shared, is not implemented.

Inference Lab increment

The Lab now has:

• chain_hashes(scope, tokens, block_size), and the deliberately wrong unchained_hashes, so a test can show why chaining is required.
• PrefixIndex: longest_prefix, adopt, register, evict, ensure_free, with reference counting through Chapter 14’s allocator.
• prefill_with_prefix_cache, which prefills only the uncached suffix of a prompt on the real model.
• TokenLevelCache, agent_workload and simulate_hit_rate, a hit-rate model with scope and capacity as parameters.
• auc, ttft, sidechannel, hitrate and exactness commands.

The rule this adds to the cache rules of Chapters 10 and 14: a cache entry is shared only within a scope that contains everything that determined it, and every scope boundary is a decision about whom the cache may inform. A speed-up that depends on sharing and a privacy guarantee that depends on not sharing are the same dial.

What you should now be able to do

• Say what determines a cached key and value, and write the scope for a given deployment.
• Explain why block keys must be chained to their history and what goes wrong without it.
• Implement lookup, adoption, registration and eviction on a reference-counted block pool.
• Predict, from a model’s prefill time and launch floor, whether caching a prefix of a given length will save anything.
• Estimate a hit rate as a function of cache size, workload and isolation.
• Measure whether a cache leaks through timing, and describe what a scope does and does not prevent.
• Read a provider’s caching documentation as an interface description, not as evidence of an implementation.

Key terms

• Prefix cache: stored keys and values for prompts that start the same way.
• Chained hash: a block key that depends on the previous block’s key.
• Scope: everything but the tokens that must match for state to be shared.
• Salt: a per-request or per-tenant value placed in the scope to prevent sharing.
• Hit rate (token): the fraction of prompt tokens served from the cache.
• LRU: evicting the least recently used entry first.
• Timing side channel: information leaked through how long something takes.
• Cache breakpoint: in a provider API, the place in a prompt up to which the caller asks for caching.

Reader exercises

1. Add the keys and values of generated tokens to the cache at the end of a request. How much does the 81% ceiling rise?
2. Implement vLLM’s refinement: when a request finishes, release its blocks so that the last one is evicted first. Does it change the hit rate on the agent workload?
3. Change the workload so that every tenant uses the same single system prompt. What is the hit rate with a scope per tenant, and with a shared scope, at 4,096 tokens?
4. Repeat the side-channel experiment with another process generating load. How many probes does the attacker now need for 95 per cent confidence?
5. Make the scope include the manifest hash of Chapter 6. Write a test in which a different quantisation scheme does not hit.
6. Add a time-to-live to entries. How does the hit rate change for sessions with long pauses?
7. Compute, for a 7-billion-parameter model with a 2,000-token system prompt, the prefill time a hit saves, using the roofline of Chapter 11.

References

• SGLang: Efficient Execution of Structured Language Model Programs — organises reusable prefix state as a radix tree (RadixAttention).
• vLLM: Automatic Prefix Caching — block hashing with parent chaining, full blocks only, SHA-256, LRU eviction and an optional cache salt for isolation; read on 21 October 2026.
• Claude prompt caching — breakpoints, lifetimes, matching rules, usage fields and isolation scope of a provider cache, as an interface; read on 21 October 2026.

Evidence in this chapter

Claim	Category
Keys and values at a position depend only on the tokens up to it, the artefact and the position	Established background; Chapter 4’s exact causal test
Block keys must include the parent’s key	Established design (vLLM’s documentation); the unchained key’s failure shown by test and by Chapter 10’s errors
Prefix reuse reproduces a full prefill (worst logit error 7.6e-5, same next token)	Book result, eight requests
A hit costs about 22 ms regardless of prefix; the saving is 0 at 128 tokens and 35 ms at 768	Book result, 15 trials per cell
With a shared scope, timing separates hits from misses completely (probability 1.00); with a separate scope it does not (0.55)	Book result, 40 trials, best-case conditions
Isolation lowers the hit rate by 2.5× at 4,096 tokens of cache and by 3 points at 65,536	Book result of a synthetic workload model
The workload model understates reuse because generated tokens are not cached	Known limitation of the model
A provider’s cache exposes breakpoints, lifetimes and usage accounting; its internals are not described	Reported by the provider’s documentation, as an interface
Poisoning, denial of service, deletion, multimodal keys, real traces, timing under load	Not measured

Implementation status

• Manuscript: content/books/inference/16-chapter.md
• Runnable examples: examples/ch16/prefix_cache.py (original scoped-dictionary simulator), examples/ch16/block_prefix_cache.py (block index, hit-rate model and experiments)
• Tests: tests/test_ch16_prefix.py, tests/test_ch16_block_prefix_cache.py (chained and unchained keys, hit with exact logits, scope isolation, the final-token rule, reference counts, eviction safety and complete return of blocks, the hit-rate model’s monotonicity and isolation effect, AUC; all on a small model, no GPU needed)
• Evidence: research/ch16/sources.md, research/ch16/results.json, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about provider internals, large models, real traces or attacks beyond the one measured. Chapter 17 turns from reuse to speculation: if scoring a known sequence is cheaper than producing it (Chapter 5), a cheap guess of the next few tokens can be checked in one pass.