← Inference From First Principles

Key-Value Caching

A KV cache written by hand into GPT-2: why it is exact, what it saves, what it costs in memory, and how it fails when its state is wrong.

I deliberately reused a key-value cache built for one prompt while continuing a different one. The code ran without an error, but its next-position logits differed from a correctly recomputed reference by a median maximum absolute error of 11.6 across twenty random trials, reaching 28.7 in the worst case. The resulting argmax agreed with the reference in only 8 of 20 trials. In a second experiment, I formed the prefix cache after multiplying one layer’s MLP output-projection weights by 1.02, then continued with the original weights. The median maximum absolute logit error was 0.042, but the argmax still agreed in all twenty trials.

The first failure would be easy to notice if the reference answer were available. The second could slip through a spot-check that compares only selected tokens. Both come from the same mistake: a KV cache is derived state, valid only under the computation that produced it. This chapter builds two caches, tests their numerical agreement with recomputation, and examines what happens when their state is reused incorrectly. It also separates a safe-reuse policy from what the current example backend actually checks.

Chapter 5 counted 1,040 token positions processed by a naive 32-token generation from a 17-token prompt, compared with 48 positions when the prompt is processed once and each subsequent input token once. Chapter 9 used the library’s KV cache as a black box. This chapter implements the missing mechanism, validates it against uncached generation and measures its costs in time, tensor storage and allocator behaviour.

Where this sits

Chapter 4 established the causal dependency: an earlier position cannot attend to tokens that come later. In one test the earlier logits were unchanged at the recorded precision; this does not guarantee bitwise equality across kernels. Chapter 5 derived the no-reuse position count without measuring a cached alternative. Chapter 8 measured cache-shaped attention in isolation and found high nominal bandwidth utilisation at larger batches, not a universal bandwidth bound. Chapter 9 timed prefill and cached decode through the library. Here we build our own cache and test its contract.

The Inference Lab increment is a pair of cache implementations and forward_cached, a GPT-2 forward pass that reads and updates cache state. CachedBackend then hides this state behind Chapter 6’s next_logits(ids) interface. It checks whether the requested ids extend the token prefix remembered from the last call and resets on a mismatch. That is a useful first guard, not yet the full state-identity contract: the backend does not compare the manifest hash or detect changes to weights or configuration on every call. A production interface must define and enforce those additional identities before it can claim safe reuse across model changes or requests.

This mechanism matters beyond a single text-completion loop. Long agent sessions can avoid recomputing an unchanged prefix, and requests with a shared system prompt may reuse its cached state when model identity and prefix identity both match. The experiment later in this chapter verifies two independent continuations from copies of one prefix cache; it does not yet implement shared cache blocks. Chapters 14 and 16 take up allocation and sharing. Changing a system prompt, model, adapter, tokenizer mapping or relevant positional configuration must invalidate any derived state that is no longer valid.

Why the cache is exact

Causal attention ensures that the hidden state at position \(i\) depends only on its visible prefix, assuming the same model parameters, positional rules, attention mask and execution semantics. Each layer’s key and value at that position can therefore be reused when the computation is extended without changing its existing prefix. The query for position \(i\) has already been used to form that position’s attention output; future positions need its key and value, not its query. This explains the cache at the mathematical level. Floating-point implementations may still differ slightly because a cached one-token operation and an uncached matrix operation use different reduction orders.

For each new input position, a decoder computes its query, key and value, appends the new key and value at every layer, and attends over the earlier entries plus the new entry. At a single-token step with a cached prefix of length \(T-1\), attention uses a \(1\times T\) score row per head. Cached decoding remains sequential across generated tokens, and its attention work and KV traffic still grow with context length. The cache removes redundant computation of the old hidden states; it does not eliminate attention over the past.

For prompt length \(P\) and \(n\) newly selected tokens, where prefill selects the first token and only the preceding \(n-1\) generated tokens need to be fed back, the position count is:

Loop Positions pushed through the model
No reuse \(nP + n(n-1)/2\)
With a cache \(P + n - 1\)

With \(P=17\) and \(n=32\), full-prefix recomputation processes 1,040 positions, while prefill plus 31 cached input steps processes 48. Their ratio, \(1040/48\approx21.7\), counts positions, not elapsed-time speedup. CachedBackend reports 48 positions per 32-token request in the recorded engine experiment, although a real device’s time also depends on matrix shapes, kernel launches and cache traffic.

A KV cache does not make total computation independent of context. Each new position still traverses the network’s layers, and its attention must access the relevant cached keys and values. The cache avoids recomputing earlier positions’ attention outputs, projections and feed-forward layers. Its benefit is consequently large when that redundant work was expensive and potentially small when launch or dispatch overhead dominates the short-context step. The precise speedup is an empirical question.

KV cache principle

What is in it, and how big

For each layer, ConcatCache stores keys and values as tensors with shape \([B,h_{kv},T,d_h]\). Here \(h_{kv}\) is the number of key/value heads—12 for GPT-2, but potentially fewer than the query-head count in grouped-query or multi-query models. Ignoring alignment and allocator overhead, the storage required for the active cache is:

\[ \text{bytes} = 2 \cdot L \cdot h \cdot d_h \cdot T \cdot s \cdot B \]
where \(L\) is the layer count, \(h_{kv}\) the number of KV heads, \(d_h\) the head dimension, \(T\) the active prefix length, \(s\) the element size in bytes and \(B\) the batch size. The table checks the formula against the sum of **live `ConcatCache` tensor sizes** (`numel × element_size`) in the recorded GPU experiment. These are logical tensor-storage counts, not independently measured physical memory traffic or peak device residency. A `StaticCache` allocates for its configured capacity even before every slot is used.
Precision Active length Formula (bytes) Concat tensor bytes
fp32 128 9,437,184 9,437,184
fp32 1,024 75,497,472 75,497,472
fp16 128 4,718,592 4,718,592
fp16 1,024 37,748,736 37,748,736

The formula agrees with the four recorded tensor-size counts. At \(T=1024\), the active KV tensors occupy 72 MiB in fp32 or 36 MiB in fp16 for one GPT-2 sequence. This is linear in context length, batch size and KV-head count. It excludes the model, temporary attention tensors, metadata, alignment, unused static capacity and allocator reservations.

The 124,439,808 GPT-2 parameters occupy about 237 MiB if stored as fp16 tensors. A full-length fp16 KV cache is 36 MiB per sequence, before other allocations. The script observed roughly 10,069 MiB of free device memory at one point and calculated an illustrative upper-budget count of 273 full-length caches after subtracting the fp16 model size. That is not a tested batch capacity: it ignores activations, other process allocations, allocator fragmentation, runtime workspace and whether the memory observation and model residency share the same conditions. Its purpose is to show scale. Even for this small model, several long-context caches can cost as much as the weights themselves.

Writing it

Three changes make the Chapter 4 forward pass cache-aware; a fourth obligation governs when that cached state may be reused.

Positions. With \(t\) new input tokens, the learned position embeddings are indexed at cache.length ... cache.length + t - 1. Starting again at zero would assign the wrong positions. GPT-2 also has a fixed learned position table, so an implementation must reject a context that exceeds its configured maximum before writing state.

The mask. A chunk of \(t\) new queries attends to \(\text{total}=\text{cache.length}+t\) keys. Query index \(i\) within the chunk has absolute position \(\text{total}-t+i\), and may attend only to keys through that position. The implementation masks the upper triangle with diagonal=total - t + 1. For \(t=1\), no earlier key is future-visible; for chunked prefill the offset remains essential.

Attention and state advancement. Each layer’s cache.update(layer, k, v) adds the newly computed keys and values and makes the complete visible prefix available to attention. After the layers complete, cache.advance(t) increments the shared length once. State validity is a separate obligation: the caller must establish that the existing tensors represent the requested prefix under the same model and positional rules. This teaching implementation does not offer transactional rollback if a forward pass fails after some layers have already written their buffers.

x = wte[ids] + wpe[arange(offset, offset + t)]
for layer in range(n_layer):
    ...
    keys, values = cache.update(layer, k, v)          # [B, h, length + t, d_h]
    scores = q @ keys.transpose(-2, -1) / sqrt(d_h)
    scores = scores.masked_fill(triu(ones(t, total), diagonal=total - t + 1), -inf)
    ...
cache.advance(t)

There are two intentionally simple storage strategies behind update:

  • ConcatCache grows each layer’s key and value tensors with torch.cat. It is easy to inspect, but allocates and copies when a suffix is appended. This resembles a simple dynamic-cache strategy; it is not a claim that every production or current library cache uses the same allocation path.
  • StaticCache allocates tensors for max_len once and writes slices in place. It returns views only through the active end position, so pre-existing values outside that range remain logically inaccessible. reset() sets the active length to zero; it does not zero the storage. Its nbytes() describes allocated capacity, whereas used_bytes() estimates the active part.

The static design’s reset deserves more than an assertion that old bytes are harmless. In a small-model regression test, a previously used static buffer is reset, loaded with a different prefix and compared with a fresh buffer; the resulting logits match exactly in that test. This supports the active-length invariant for the exercised path, not a guarantee against every aliasing or concurrency error. A future test should deliberately interrupt an update halfway through and check that the next request cannot see partial state.

Concat vs Static cache

It is the same function

The first acceptance condition is numerical agreement, not mere speed. In the recorded 17-token-prompt experiment, I generated 48 tokens greedily with no reuse, with each hand-written cache design on the CPU and GPU, and with the library cache on the GPU. All recorded token sequences matched the uncached reference. This is encouraging evidence for these inputs and execution paths; it is not a proof of equality for all prefixes or decoding methods.

Tokens identical to no-reuse?
Concat cache, GPU and CPU yes, all 48
Static cache, GPU and CPU yes, all 48
Library cache, GPU yes, all 48

The logits give a more sensitive comparison than the selected tokens. For each of 48 positions, the parity command compared cached logits with full recomputation over the same growing sequence. On the GPU, the maximum absolute error for a position had median \(6.1\times10^{-5}\) and worst value \(6.7\times10^{-4}\); the CPU worst was \(5.3\times10^{-4}\). A separate library comparison at the original prompt differed by about \(6.1\times10^{-5}\) on the GPU and \(9.2\times10^{-5}\) on the CPU. Different shapes and floating-point operation orders can change the last few bits. Matching argmaxes across 48 steps does not rule out a rare near-tie causing a different token elsewhere, and an absolute-error threshold should be accompanied by a relative-error or decision-margin analysis for other models.

The checked-in tests exercise both cache designs on a tiny, randomly initialised GPT-2: greedy generation against recomputation, logits against the library, chunked versus one-shot prefill, cache sizes, static overflow and reset behaviour, an intentionally wrong-prefix result, branch independence and the backend’s prefix-hit/miss accounting. These tests do not execute the real-model measurement commands or verify the effect of hot-swapping weights after a cache is created. The recorded GPT-2 measurements in research/ch10/results.json are separate evidence.

What it saves

The next experiment times a single new input position after a prefix of length \(T\). For the no-reuse case the same hand-written forward pass processes the entire prefix plus the new token with a fresh cache. In the cached cases the prefix is prefilled once outside the step measurement and subsequent steps extend a growing cache. The table reports two runs, with medians taken over multiple steps inside each run. It measures steady-state step cost, not the total cost of building the cache or completing a request.

Context GPU: no cache GPU: concat GPU: static CPU: no cache CPU: concat CPU: static
16 9.2 – 10.6 ms 9.1 ms 8.4 – 8.8 ms 77 – 79 ms 33.5 ms 29 – 31 ms
128 13.0 – 14.1 ms 9.2 – 11.7 ms 9.6 – 11.4 ms 279 – 299 ms 36 – 42 ms 33 – 34 ms
512 35.6 – 38.4 ms 9.1 – 12.1 ms 9.4 – 9.8 ms 1,141 – 1,185 ms 42 – 47 ms 34 – 49 ms
960 81 – 96 ms 8.4 – 16.5 ms 8.6 – 12.6 ms – – –

Ratios of uncached to concat-cached step time within each recorded run:

Context GPU CPU, one thread
16 1.0 – 1.2× 2.3×
128 1.2 – 1.4× 7.2 – 7.8×
512 3.2 – 3.9× 24 – 28×
960 4.9 – 11.5× –

CPU reuse pays back early in this experiment. At a 16-token context, the measured step was about 2.3 times faster with concat reuse; by 512 tokens the observed range was 24–28 times. That is consistent with removing repeated matrix work on the one-thread CPU. The benchmark did not instrument individual CPU bottlenecks, so calling every case strictly arithmetic-bound would go beyond the evidence.

On this GPU, reuse brings little benefit for very short contexts. At 16 tokens, cached and uncached steps differed by no more than roughly 20 per cent in the two recorded runs. That is consistent with a substantial fixed launch/dispatch cost, but this chapter did not isolate that cost with a profiler. At context 512, reusing the cache was 3.2–3.9 times faster; at 960, the measured range was 4.9–11.5 times. Those are within-run step speedups. The cache’s own step time varied from about 8 to 16 ms across the tested contexts and runs.

The largest spread is at 960 tokens: the concat step ranged from 8.4 to 16.5 ms between the two runs. That variation is too large to support a general performance ranking of concat against static. A controlled repetition with device clocks, warm-up, equivalent cache occupancy and more samples is needed.

Inside the engine

The Chapter 6 engine then provides a request-level comparison: 32 greedy tokens after a 17-token prompt, using the uncached library backend or either cached hand-written backend. This is an integration test of the engine and its accounting, not a controlled estimate of the cache’s speedup, because the forward implementations differ.

Device Backend ms per output token Output tokens/s Same ids as uncached library? Positions per request
CPU library, no reuse 118.8 8.4 reference 1,040
CPU hand-written, concat 33.5 29.8 yes 48
CPU hand-written, static 35.0 28.6 yes 48
GPU library, no reuse 8.5 117 reference 1,040
GPU hand-written, concat 9.9 101 yes 48
GPU hand-written, static 10.1 99 yes 48

The CPU hand-written cached engine finished about 3.5 times faster than the uncached library engine in this particular 32-token test. The GPU hand-written cached engines were slower than the uncached library path, roughly 9.9–10.1 ms versus 8.5 ms per token. The likely contributors include extra eager-PyTorch operations and fixed dispatch overhead, but none was isolated here. The preceding within-implementation step comparison is better evidence for the benefit of caching itself. At the request level, the result also depends on prefill, sampler and host-side overhead.

Absolute times should not be joined across measurement sessions as though the device state were controlled. Chapter 9 recorded roughly 11 ms for a cached library decode step; this chapter recorded roughly 5–8 ms under related but not identical measurement conditions. The additional check found host selection and readback accounted for about a millisecond, not the whole difference. The remaining cause was not established. The safe conclusion is a speedup within a matched run, with the exact timing boundary and environment recorded.

What it costs in memory

Growing and preallocated caches put different pressure on the PyTorch allocator. The memory command prefilled 16 random tokens, then extended each cache with 944 further one-token inputs to reach 960 active positions at batch sizes 1 and 8 in fp32. It did not generate meaningful text; the appended token inputs were synthetic. The benchmark reports active KV tensor size, peak PyTorch allocation above a baseline captured after model loading, and total allocator reservation at the end. These are different measures; the reserved-memory column includes allocations unrelated to the KV tensors.

Batch Design Active KV bytes at 960 Peak extra PyTorch allocated PyTorch reserved at end
1 concat 67.5 MiB 71.3 MiB 636 MiB
1 static 67.5 MiB (72 MiB capacity) 73.1 MiB 610 MiB
8 concat 540 MiB 563 MiB 2,926 MiB
8 static 540 MiB (576 MiB capacity) 588 MiB 1,108 MiB

The peak additional allocated memory is close to the active cache size for both designs. With concat growth, temporary copies are created one layer at a time rather than doubling every layer simultaneously. The sharp difference is in allocator-reserved memory at batch 8: 2,926 MiB for concat versus 1,108 MiB for static, a difference of 1,818 MiB. Reserved memory is held by PyTorch’s caching allocator and may be available for reuse by PyTorch; it is not, by itself, proof of irrecoverable fragmentation, live-tensor demand or the same amount of physically resident device memory. Repeatedly requesting ever-larger tensors plausibly contributes to the observed reservation gap, but this experiment did not instrument allocator block reuse.

This single batch-8 result motivates explicit allocation policies, not a universal 1.8 GiB penalty for dynamic caches. Static allocation trades an upfront capacity commitment for fewer resizing allocations; concat growth uses only the active length in its live tensors but can churn the allocator. Chapter 14 explores blocked allocation and sharing. To attribute the reservation gap rigorously, the Lab would need an allocator snapshot, controlled cache-clearing conditions and repetitions across batch and context sizes.

State identity

The model defines the validity contract; the current backend enforces only part of it. Cached keys and values depend on the precise earlier token ids and on the computation that produced them: model weights, active adapters, tokenizer mapping if text is being tokenized, positional scheme, attention masks and relevant execution configuration. In CachedBackend.next_logits(ids), the implemented reuse test is self.cache is not None, 0 < len(self.tokens) < len(ids) and ids[:len(self.tokens)] == self.tokens. It resets otherwise. The test does not check the cache’s manifest or compare a stored model fingerprint at every call, and it does not separately assert cache.length == len(self.tokens). In the recorded engine experiment each of five GPU requests reset once and reused the following 31 prefixes, giving 5 resets and 155 reuses over 160 calls.

The helper prefix_key(manifest_sha256, token_ids) computes a SHA-256 digest from a manifest hash and the ordered token ids. It is useful as an example of an identity key, and tests confirm that changing those inputs changes the sample outputs. It is not wired into CachedBackend and does not implement a shared prefix-cache index. Nor does a hash automatically guarantee that the manifest covers live adapter state, positional rules, software versions and any mutable weights. Making the identity policy enforceable requires defining those dependencies, freezing or versioning them, validating the cache’s active length and invalidating state when any relevant dependency changes.

What goes wrong

The hazard command exercises four deliberately invalid reuse cases and two controls, each on 20 seeded trials using a random 24-token prefix and one supplied continuation token. Its reference is a fresh recomputation of the logits after that supplied token. The final column therefore means argmax agreement with the reference computation, not correctness against a ground-truth next token in a dataset.

Cache scenario Median max-absolute logit error Worst max-absolute error Argmax agrees with reference
Correct reuse 0.00003 0.00005 20/20
Built for another prompt 11.6 28.7 8/20
Wrong position offset 54.9 72.1 1/20
Static buffer reused without resetting length 5.7 17.3 14/20
Static buffer reused after resetting length 0.00003 0.00005 20/20
Prefix computed using perturbed weights 0.042 0.158 20/20

A stale prompt cache, incorrect position offset or forgotten static reset produced substantial logit errors in this small sample. Some erroneous computations nonetheless kept the same argmax: 8 of 20 for a different prompt and 14 of 20 for a missed reset. These fractions are not comparisons with chance accuracy, because the model’s argmax distribution is not uniform over its vocabulary. The regression lesson is simpler: testing only the chosen token would miss several demonstrably invalid cache states.

The perturbed-weight experiment is more subtle. The command multiplies one layer’s MLP output-projection weight matrix by 1.02 while constructing the prefix cache, then resumes using the unmodified model. It observes a median maximum absolute logit difference of 0.042, yet all 20 argmaxes agree. This imitates one aspect of a model or adapter change; it is not an actual adapter-swap measurement. Also, the 0.042 number is an absolute error in logits, whereas Chapter 7’s \(3.6\times10^{-4}\) figure is a relative error of matrix products. They cannot be compared directly to declare the cache numerically acceptable. The defining problem is that the two portions of the computation used different weights.

A correct response to this class of error is state invalidation, not an arbitrarily loose logit tolerance. In a complete implementation, the cache’s identity must bind the prefix to a frozen or versioned model and all relevant configuration, and the backend must check that identity before reuse. This chapter defines a manifest-based hash helper but does not yet enforce it in the backend. A real adapter change or in-place weight mutation must therefore be handled by an explicit invalidation hook or by implementing the full identity check; simply trusting the current prefix comparison would leave the defect possible.

State identity and failure modes

Sharing a prefix

Two continuations can safely start from the same validated prefix state. In the recorded example, a 512-token prefix produced a 36 MiB fp32 concat cache. I used fork() to clone that cache into two separate sets of tensors, continued each with a different supplied token and compared both results with independent full recomputation. Each branch differed by at most \(6.1\times10^{-5}\) in logits; the parent cache’s length remained 512. This verifies isolated branch correctness for one example. It does not save memory by sharing physical KV blocks—fork() copies the tensors—and it does not measure time saved by a serving system. Chapters 14 and 16 address actual shared-block storage and copy-on-write.

What the numbers do not show

  • Small model and specific hardware. The short-context GPU result is consistent with a launch/dispatch floor; the individual bottlenecks were not directly profiled. Larger models and other hardware may behave differently.
  • Two forward implementations. The within-implementation step comparison isolates reuse more cleanly than the engine table, which compares hand-written cached code with an uncached library backend.
  • Timing variability. Absolute step times drifted significantly between sessions; two runs per context are insufficient for a general ranking.
  • Tensor bytes versus allocation. Formula checks count tensor storage; peak allocation and reserved allocator memory answer different questions. The memory benchmark used batches 1 and 8 only.
  • Random-prefix hazards. Twenty trials with synthetic tokens, one supplied continuation token and one deliberately perturbed weight matrix do not establish rates under real workloads or adapter changes.
  • Incomplete identity enforcement. prefix_key is not used by CachedBackend, which checks only the token prefix. Mutable model state, mask/position changes, partial updates and concurrent calls require additional safeguards.
  • No production cache manager. Eviction, paging, quantisation, shared physical blocks, rollback and ragged batches are outside the tested scope.
  • Illustrative capacity only. The 273-sequence calculation ignores activation/workspace memory, allocator behaviour and other applications.

Inference Lab increment

The Lab now has:

  • ConcatCache and StaticCache, with update, advance, reset, fork and nbytes; StaticCache also exposes used_bytes(). The cache_bytes(...) helper computes active tensor-storage requirements.
  • forward_cached(ids, weights, config, cache) for full prefill, chunked prefill and one-token continuation, with learned-position offsets and an offset causal mask.
  • CachedBackend, which plugs into the Chapter 6 engine, checks whether incoming token ids extend its remembered prefix, resets otherwise, and counts calls, reuses, resets and processed positions. It does not yet bind the active cache to a checked manifest key.
  • prefix_key(manifest_sha256, token_ids), a deterministic identity-key helper that is currently separate from the backend.
  • Reproducible parity, speed, memory, hazards and engine experiments, with recorded results and tests that can be run without downloading GPT-2.

The acceptance rule is stricter than the current demo implementation: reuse cached state only if the stored token prefix, cache length and every relevant model/configuration identity are valid for this request. The simple backend covers the first condition through a prefix check. The next engineering task is to persist and verify the full identity, invalidate on changes, and make updates failure-safe. A faster path whose cache provenance cannot be established is not yet a safe optimisation.

What you should now be able to do

  • Derive why causal attention permits reuse of previous keys and values, while floating-point results need not be bit-identical.
  • Calculate active KV storage, distinguishing query heads from KV heads, and separate tensor size from allocator reservation.
  • Count the positions processed by prefill plus cached decode versus full-prefix recomputation, without mistaking the ratio for speedup.
  • Implement position offsets, chunk-aware causal masking and consistent per-layer advancement.
  • Reproduce token-level parity and compare logits with a stated numerical tolerance.
  • Explain the conditional CPU and GPU speedups using matched implementations and measurement boundaries.
  • Distinguish concat growth from static capacity allocation and interpret the allocator metrics cautiously.
  • Identify stale-prefix, wrong-position, missed-reset and changed-weights hazards, and explain why matching argmaxes can conceal them.
  • Design a manifest-aware identity and invalidation check beyond the current token-prefix-only backend.

Key terms

  • KV cache: stored keys and values of earlier positions at each layer, reusable while the earlier computation remains valid.
  • KV head: an attention head that stores keys and values; there may be fewer KV heads than query heads in GQA or MQA.
  • Static cache: capacity preallocated in fixed tensors and filled in place.
  • Concat cache: storage grown through concatenation as positions arrive.
  • State identity: the prefix and complete set of model/configuration dependencies that produced a cache.
  • Invalidation: discarding or resetting state when its identity no longer matches.
  • Fork: making independent continuation state; this chapter’s fork() clones tensor contents, not shared physical blocks.
  • Allocator-reserved memory: memory held by a framework’s caching allocator, distinct from memory occupied by live tensors.
  • Prefix key: a hash of a manifest identifier and ordered prefix tokens; in this Lab it is a helper, not an enforced backend check.

Reader exercises

  1. Remove the total - t + 1 causal-mask offset and measure the first divergent logit, not just the first divergent selected token. Explain the position that became wrongly visible.
  2. Compute the fp16 cache size for 32 layers, 32 KV heads, head dimension 128 and context 8,192. Repeat with eight KV heads, and state what a 24 GiB budget does not tell you about feasible batch size.
  3. Extend CachedBackend to persist (manifest_sha256, token_prefix, active_length) alongside the tensors. Add tests where the ids do not change, but the backend’s weights or adapter identity does; ensure the old state is rejected.
  4. Inject a failure in layer 2 of forward_cached after an earlier layer has updated its cache. Show whether the next call can reuse inconsistent state, then implement reset-on-failure or transactional advancement.
  5. Repeat the allocator experiment at batch 32, record allocated and reserved memory separately, and capture an allocator snapshot before calling the difference fragmentation.
  6. Make a rollback operation that truncates a cache to length \(k\); test both cache designs, update the stored prefix identity and reject out-of-range rollback.
  7. Modify greedy_cached so it does not execute an unnecessary extra cached forward pass after selecting the final requested token. Check its final cache.length, especially at the model’s context limit.
  8. Compare two different model-weight perturbations using both logit max-error and argmax decision margins, and explain why matching tokens is insufficient.
  9. Replace cloned fork() branches with a read-only shared prefix and copy-on-write suffix. Measure real memory sharing rather than just numerical parity.

References

Evidence in this chapter

Claim Evidence classification
Causal attention permits reuse of unchanged prefix KV state Established model property, subject to model/configuration identity
Active storage is \(2Lh_{kv}d_hTsB\) bytes Arithmetic; four concat tensor-size checks agree
Concat, static and library cached generation matched 48 uncached greedy tokens on tested paths Book result, one GPT-2 prompt, CPU/GPU as recorded
Per-step maximum absolute logit differences: median about \(6.1\times10^{-5}\), worst GPU \(6.7\times10^{-4}\) Book result, one prompt and 48 steps; numerical, not bitwise equality
32-token request: position count 1,040 no reuse versus 48 with cache Analytic count and book backend accounting
CPU cached step at context 512: about 24–28× faster than recomputation Book result, two runs, same hand-written forward
GPU cached step at context 512: about 3.2–3.9× faster Book result, two runs; short contexts show little gain
Cached hand-written engine slower than uncached library on GPU at 17–48 positions Book result, mixed implementations; not causal attribution
Batch-8 concat run reserved 1,818 MiB more PyTorch allocator memory than static Book result, one configuration; cause not directly profiled
Four incorrect reuse scenarios caused logit deviations Book result, 20 random trials each; last column is reference argmax agreement
A 2% perturbation of one MLP projection changed logits without changing observed argmaxes Book result, single synthetic weight perturbation; not an adapter swap
Forked continuation caches matched reference logits and kept parent length Book result; deep copies, not physically shared KV storage
Token-prefix validation in CachedBackend Implemented and tested; full manifest identity not enforced
273 full-length fp16 caches could fit in a memory budget Illustrative arithmetic, not a measured operating capacity
Paged cache management, physical prefix sharing, eviction, rollback, adapter invalidation NOT_IMPLEMENTED / NOT_MEASURED in this chapter