PagedAttention
A KV cache in fixed-size blocks: how much memory contiguous allocation wastes, what block tables, reference counts and copy-on-write buy, and what greedy admission costs.
A simulated server has a KV budget equivalent to 32,768 token slots. A policy that reserves 1,024 slots per sequence can admit at most 32 sequences. During the measured steady portion of a synthetic chat-like workload, only 33 per cent of those reserved slots held valid tokens. The rest was capacity reserved in advance, not useful KV state.
With the same budget, an allocator that grants 16-token blocks as sequences grow held about 92 concurrent sequences with a 5 per cent admission watermark, or 96 with no watermark. Valid tokens occupied roughly 98 per cent of the blocks allocated to resident sequences. That is block-packing efficiency, not 98 per cent utilisation of the entire KV pool. Under the five-per-cent watermark, the simulated 3,000-request run used 5,611 scheduler steps against 15,260 for maximum reservation: about 2.72 times fewer steps, before accounting for GPU execution and prefill.
Now remove the admission watermark. New prompts can fit even when little room remains for their continuations. In the chat-like simulation the scheduler preempted a sequence 2,094 times. The run took 5,378 simulated scheduling steps; the report’s 13,754 figure adds a further four-step penalty for each preemption. This is substantially worse than the watermark policy, although still below maximum reservation’s 15,260 steps, and worse than the declared-limit policy’s reported 10,637. Those are modelled step counts, not observed latency or throughput. There is also an important flaw in the declared-limit comparator: some sampled outputs exceed its declared allowance. We will keep that caveat visible rather than silently treating the ranking as validated.
Paging solves a memory-layout problem; admission and eviction determine how much of that solution becomes useful throughput. This chapter constructs a reference paged KV cache for GPT-2, measures the cost of gathering its blocks into contiguous attention inputs, and runs a synthetic allocator/scheduler experiment. Neither that gather-based implementation nor the simulator is a measurement of a production PagedAttention kernel.
Where this sits
Chapter 10 compared growing and preallocated caches. For eight sequences at 960 tokens, the measured PyTorch allocator reserved 2,926 MiB with concatenation versus 1,108 MiB with static buffers; the logical cache held 540 MiB. Reservation is not itself a direct measurement of external fragmentation, but it shows why allocation behaviour matters. Chapter 8 calculated where KV bytes overtake weight bytes in an idealised batched decode. Chapter 9 showed how two requests compete for a shared execution timeline. Here the unit of management becomes a set of independently growing KV sequences backed by one physical block pool.
The Inference Lab increment is the block layer: reference-counted BlockAllocator, BlockTables with forks and copy-on-write, a PagedKV tensor pool, ragged-batch GPT-2 decoding, and a deterministic simulator comparing reservation and on-demand allocation policies. These are separate kinds of evidence. The allocator tests establish selected safety properties; the GPU experiment checks one gather-based implementation; the simulator tests assumed arrivals and sequence lengths. An allocator should ultimately be judged by correctness, reserved and useful bytes, admitted concurrency, preemptions, and end-to-end service objectives, not a single utilisation percentage.
For the Pi series, agent conversations, tool responses and shared system prompts create contexts of variable length and often repeated prefixes. A block pool can store those contexts without reserving every possible future position, and reference counts can represent shared prefixes without immediately copying them. Actual reuse also needs the cache-identity validation established in Chapter 10; a matching block table alone does not prove that two requests used the same model, positions or tokenisation. Chapter 16 extends this mechanism into prefix-cache lookup and eviction.
Where contiguous allocation wastes memory
A cache stored as one contiguous region per sequence can waste memory in three ways.
Advance reservation. A simple contiguous-cache design may reserve space for the entire model context or a declared generation limit so that growing the sequence cannot fail. That space is capacity committed to this sequence, not necessarily a tensor containing useful keys and values. Contiguous buffers can also grow or be reallocated; advance reservation is a design choice, not a logical requirement of contiguous storage.
Internal fragmentation. A fixed-capacity allocation can contain positions that have not been filled. In a paged pool the equivalent overhead is usually confined to the partly filled last block of each live sequence, unless shared-block or preallocation policies add more.
External fragmentation and allocator overhead. Variable-sized allocations can leave free regions that do not satisfy a later large allocation, while caching allocators can retain previously allocated blocks. Chapter 10 observed a large difference in reserved PyTorch memory, but did not isolate fragmentation from allocator caching, temporary buffers and other sources of reservation. Equal-sized physical KV blocks avoid the need for contiguous per-sequence space at that granularity.
The simulator principally models advance reservation and last-block slack. It does not simulate a GPU memory allocator, kernel execution, request arrival times, host transfer, or external fragmentation. It keeps a saturated queue of 3,000 requests, a budget of 32,768 KV-token slots, and lognormal prompt/output lengths capped by a 1,024-token context. Each resident sequence advances one generated token per scheduler step, and prompt prefill consumes no simulated time. The three workload definitions are parameters, not observed chat traces:
| Workload | Mean prompt | Mean output | Mean final length |
|---|---|---|---|
| Chat-like | 217 | 154 | 371 |
| Long prompt, short answer | 627 | 47 | 674 |
| Short prompt, long answer | 47 | 359 | 406 |
The input distributions are specified by medians; the means in the table are estimates from 5,000 separately sampled requests. For a reservation comparator, reserve_max commits 1,024 slots per sequence. reserve_declared commits min(1,024, prompt + 512) slots, but the generator does not cap every sampled output at 512. In the seeded 3,000-request samples, 92 chat-like requests and 681 short-prompt/long-answer requests need more than their declared reservation; the simulation nevertheless lets them grow. That comparator therefore needs a repaired admission/length rule before its results can support a real capacity comparison. The reported memory utilisation is sum(live logical lengths) / sum(reserved or allocated slots) over scheduler steps while requests are waiting; it is a packing ratio for resident allocations, not the fraction of the 32,768-slot pool occupied.
| Policy | Chat-like | Long prompt | Long answer |
|---|---|---|---|
| Reserve the maximum (1,024) | 33% | 61% | 30% |
| Reserve the declared limit | 48% | 64% | 55% |
| 16-token blocks, on demand | 98% | 99% | 97% |
Under these definitions, maximum reservation holds only 30–61 per cent useful tokens relative to the slots it committed; the fixed-block allocator holds 97–99 per cent useful tokens relative to the blocks it allocated. Neither percentage reports GPU capacity in use, allocation throughput, or observed requests per second. A small watermark deliberately leaves part of the pool unallocated, so block-packing efficiency can stay near 98 per cent while pool occupancy and concurrency fall. The declared-limit rows are informative about the intended comparison but subject to the output-limit defect noted above.
Paging
The paged cache decouples a sequence’s logical positions from where they sit in memory.
- A physical KV block stores a fixed number of consecutive token positions’ keys and values, for all layers in the chosen pool layout.
- A sequence’s block table maps logical block index \(i\) (token positions \(ib\) through \(ib+b-1\)) to a physical block identifier. Blocks belonging to one sequence need not be adjacent.
- An allocator grants blocks as a sequence grows and returns them when its last owner releases them. The present
PagedKVallocates the whole tensor pool up front; demand allocation applies to assigning blocks to sequences, not to allocating GPU tensor storage. - Reference counts permit multiple sequences to address the same blocks. A shared partially filled last block needs copy-on-write before a branch appends; fully occupied shared prefix blocks remain read-only.
This borrows the logical-to-physical mapping idea from virtual memory, but the implementation is not a virtual-memory system: it has no page faults or address translation by hardware. BlockAllocator.alloc() removes one identifier from a Python free list in constant time, while allocating several blocks, building tables, and copying a shared block cost additional work. Equal-sized blocks remove the contiguous-allocation requirement for sequence growth; they do not remove last-block slack, metadata costs, or the possibility of running out of free blocks. For a 1,024-token sequence and 16-token blocks, 64 idealised 32-bit identifiers would take 256 bytes. The actual Python-list bookkeeping has more overhead, and the example’s fp32 GPT-2 KV data are 72 MiB per full sequence.
What it costs in the last block
If final sequence lengths are approximately uniform modulo a block size \(b\), an occupied last block wastes an average of about \((b-1)/2\) token slots. Dividing that by mean useful length \(\bar L\) gives a convenient small-overhead approximation to the fraction of useful tokens spent on slack. For a true fraction of allocated block capacity, the denominator also includes the wasted slots, and the length distribution and arrival/departure weighting matter. The approximation is
| Block size \(b\) | Predicted waste | Measured waste |
|---|---|---|
| 1 | 0% | 0.0% |
| 16 | 2.0% | 2.2% |
| 64 | 8.5% | 8.7% |
| 128 | 17.1% | 16.0% |
| 256 | 34% | 27.3% |
The 16- and 64-token results agree with the approximation to within a few tenths of a percentage point. At 256 tokens it predicts 34 per cent while the reported loss is 27.3 per cent: the approximate denominator, sequence-length variation and the simulator’s resident-time weighting now matter. The practical result is that block size trades last-block slack against table length and kernel access granularity; it is not a universally optimal constant.
Why not use \(b=1\)? It removes final-block slack but requires a table entry per token and potentially many fine-grained indirect accesses. This implementation’s stand-alone gather benchmark shows a modest difference across block sizes for one shape; it does not measure a fused paged-attention kernel, cache locality or table-read overhead directly. Smaller blocks can also increase allocation and metadata operations. Choose block size with the attention kernel and the actual context-length distribution, not from the slack formula alone.
The real cache
PagedKV preallocates a key pool and a value pool, each shaped [layers, physical_blocks, block_size, heads, head_dim]. A physical block identifier is shared across layers. prefill_paged adapts Chapter 10’s forward_cached interface for a single sequence by gathering its blocks. forward_paged_decode processes one position for every sequence in a potentially ragged batch, gathers K/V blocks into padded contiguous tensors, and masks padding before attention. This is a working reference implementation of paged storage and bookkeeping; it does not implement the fused PagedAttention attention-access algorithm described by Kwon and colleagues. The total pool allocation stays resident even while many physical blocks are free.
It is the same function
Four prompts of 5, 19, 40 and 77 tokens were continued together for 20 steps with 16-token blocks and compared with independent Chapter 10 caches. All recorded greedy token choices matched. The largest absolute logit difference was \(1.129\times10^{-3}\); dividing by a selected large logit magnitude can suggest parts-per-million agreement, but that is not a global relative-error bound and says nothing about positions where the reference logit is near zero. The small-model CPU tests independently exercise ragged decoding, block reuse and forked branches. These establish correctness for tested cases, not for arbitrary batch order, pool exhaustion, mixed models or every mask.
Sharing and copy-on-write
A 40-token prefix fills two 16-token blocks and half of a third. Forking the sequence initially created no new physical blocks: both block tables referenced the same three blocks. When the first branch appended a token, its shared partial block was copied; the other branch subsequently wrote in place as the sole owner. There were four physical blocks after both continuations, instead of six for two independently stored 40-token prefixes. The measured maximum absolute logit differences against recomputation were \(3.05\times10^{-5}\) and \(6.10\times10^{-5}\) for the two branches.
Two complete prefix blocks remain shared in this example; only the last partial block needs a private copy on extension. It demonstrates reference-counted ownership and copy-on-write, not a measured production prefix-hit rate or a persistent cache shared across requests. One safety obligation also remains for the implementation: BlockTables.reserve performs a partial-block copy-on-write before attempting all further allocations. If a later allocation fails, the earlier copy can remain installed even though the reservation raised OutOfBlocks. The existing no-leak test covers a failed reservation without this shared-partial-block condition, so an atomic rollback regression test and implementation fix are needed.
What the gather costs
The reference implementation reads a sequence’s physical blocks into contiguous K/V tensors before performing attention; the ragged-batch version also pads to the longest sequence. A fused paged-attention kernel instead reads indexed blocks directly. Here we measured the gather-based reference path, not such a kernel. The comparison below uses the same hand-written GPT-2 weights and model forward for a static cache and a paged cache, with 15 measured decode calls per cell in each of two runs. For timing, the static cache was populated by a forward pass, whereas the paged pool’s context lengths and block tables were initialised without computing real prefix keys and values. The runs therefore test the timing of the decode path at comparable shapes, not end-to-end request generation.
| Batch | Context | Static cache | Paged (16-token blocks) | Ratio |
|---|---|---|---|---|
| 1 | 128 | 18.1 – 19.6 ms | 18.8 – 21.1 ms | 1.04 – 1.08 |
| 8 | 128 | 22.0 ms | 24.3 – 24.6 ms | 1.12 |
| 32 | 128 | 18.5 – 21.3 ms | 20.4 – 21.4 ms | 0.96 – 1.16 |
| 64 | 128 | 20.3 – 22.8 ms | 20.9 – 22.0 ms | 0.96 – 1.03 |
| 32 | 512 | 18.3 – 18.7 ms | 33.6 – 37.5 ms | 1.8 – 2.0 |
| 64 | 512 | 19.4 – 19.9 ms | 59.2 – 63.8 ms | 3.0 – 3.3 |
At short contexts and small batches, these runs show little consistent difference between the implementations; their roughly 18–25 ms step times are compatible with substantial dispatch and framework overhead, but the cause was not isolated. At batch 32 with 512 tokens, the paged path took 1.8–2.0 times as long; at batch 64 it took 3.0–3.3 times as long. The latter cache contains \(64\times512\times73{,}728\approx2.42\) decimal GB of fp32 K/V state across 12 layers (2.25 GiB). Gathering K and V into new tensors and subsequently reading them in attention increases data movement; padding, tensor layout, indexing, kernel selection and launch costs may contribute too. No hardware memory-traffic counters were collected, so the entire extra 40 ms cannot be assigned to a known number of DRAM passes.
The engineering conclusion is narrower but useful: this implementation’s gather path becomes expensive at large batch and context. The PagedAttention paper addresses that different problem by coupling the block allocator with an attention kernel that reads via the block table without first materialising a full contiguous KV cache. The experiment motivates a fused kernel, but does not measure its cost or prove that such a kernel would eliminate every overhead here.
I isolated one single-tensor gather, using eight sequences of 512 tokens, 12 heads and head dimension 64 in fp16. The gathered output contains 6 MiB of data per layer for one of K or V; reading the source and writing the result implies 12 MiB of nominal copy traffic. Physical block identifiers were either randomised or sequential. This microbenchmark differs from the complete multi-layer, K-and-V gather used during decoding:
| Block size | Gather time, scattered | In order |
|---|---|---|
| 1 | 81 – 82 µs | 80 – 81 µs |
| 16 | 78 µs | 77 – 78 µs |
| 256 | 74 µs | 74 µs |
Across these two runs, larger blocks reduced gather time modestly, from about 81–82 µs at block size 1 to about 74 µs at 256. Random physical order was within roughly 1 µs of sequential order in this indexed-copy microbenchmark. Dividing 12 MiB of nominal read-plus-write traffic by the elapsed time gives about 154–170 GB/s, an effective traffic estimate, not measured DRAM-bandwidth telemetry. Neither the block-table overhead of a fused kernel nor the locality of real attention accesses was measured.
The policy is where it succeeds or fails
On-demand allocation allows admission after the prompt’s initial blocks fit, without reserving every future generated position. A subsequent decode step may nevertheless need another block when the pool is exhausted. The simulator chooses the newest admitted resident request for preemption, returns its blocks and reinserts its original prompt and output length into the waiting queue; when readmitted, the simulated request restarts from its original prompt. The watermark rule accepts a new prompt only if a chosen fraction of the block pool remains free after admission. It is a simple capacity policy, not a measured production scheduler or an optimised eviction algorithm.
The following table reports the simulator’s steps_if_recompute_costs_4_steps: scheduler steps until all 3,000 original requests complete, plus an extra four-step penalty per preemption. The simulator already repeats lost generation work after preemption, so four steps is an additional fixed surcharge, not a measurement of full prefill or recomputation time. All rows use 16-token blocks for paging, a 32,768-token budget, saturated admission and the same seeded request stream. Lower is better under those assumptions; the declared-limit reservation baseline has the over-limit defect identified earlier.
| Policy | Chat-like | Long prompt | Long answer |
|---|---|---|---|
| Reserve the maximum | 15,260 | 4,665 | 35,223 |
| Reserve the declared limit | 10,637 | 4,536 | 19,711 |
| Paged, watermark 0 | 13,754 | 3,877 | 94,960 |
| Paged, watermark 5% | 5,611 | 3,106 | 17,752 |
| Paged, watermark 10% | 5,858 | 3,273 | 14,599 |
| Paged, watermark 20% | 6,550 | 3,671 | 14,489 |
| Paged, watermark 30% | 7,446 | 4,186 | 15,650 |
At zero watermark, the three workloads recorded 2,094, 228 and 20,809 preemptions respectively. With a 5 per cent watermark they recorded 11, 0 and 1,501; with 10 per cent they recorded 2, 0 and 613. Thus 5–10 per cent greatly reduced preemptions in all three workloads but did not eliminate thrashing in the long-answer workload. Those counts are simulation outputs, not serving traces.
With no admission headroom, paging scored better than maximum reservation in the chat-like and long-prompt workloads, but much worse in the long-answer workload after the four-step surcharge: 94,960 steps versus 35,223. The uncharged base scheduler steps for that same zero-watermark run were 11,724; therefore the apparent ranking is highly sensitive to the chosen preemption-cost model. With 5 per cent headroom, the reported counts fall to 5,611, 3,106 and 17,752 respectively. A larger watermark gives up concurrency: in the chat-like workload mean resident sequences fall from about 96 at zero watermark to about 68 at 30 per cent. The block-packing ratio nevertheless remains about 98 per cent, illustrating again why it must not be called GPU-pool utilisation.
Against maximum reservation, the reported simulated step counts at watermark 5–10 per cent improve by 2.6–2.7× (chat-like), 1.4–1.5× (long prompt) and roughly 2.0–2.4× (long answer). These comparisons exclude real prefill, kernel and request-queue costs, and the declared-limit comparator is not yet consistently enforced. Kwon et al.’s PagedAttention paper reports a 2–4× throughput improvement at comparable latency for the vLLM serving system over specific contemporary baselines; that is independent published system evidence, not confirmation of these simulated step ratios. The similarity of the numbers is not a reproduction of its result.
Several limitations are structural. The simulator has no arrival process: the queue is always backlogged, and all resident requests are advanced by one token per scheduler step. Prompt prefill has zero modelled duration. On preemption, progress is discarded and later replayed; the report can then add a fixed one- or four-step surcharge, rather than pricing the actual prefill with Chapter 9’s measured costs. A real service would also model KV memory for in-flight prefill, heterogeneous decode times, client-visible TTFT and TPOT, fairness, deadlines, and alternative eviction or host-spill policies. The aim here is to show policy sensitivity, not identify the best production watermark.
A projection, by arithmetic
For an illustrative 7B-class architecture with 32 layers, 32 KV heads and head dimension 128, fp16 KV storage per token is \(2\times32\times32\times128\times2=524{,}288\) bytes, or 0.5 MiB. A 1,024-token sequence requires 512 MiB of logical KV state. A separate, hypothetical 10 GiB KV-only budget therefore holds 20 fully reserved sequences or about 54 mean-length sequences when the assumed 98 per cent block-packing ratio is included: \(10{,}240\,\text{MiB}\times0.98/(371\times0.5\,\text{MiB})\approx54\). It is an expected-capacity ratio, not a guarantee that 54 concurrent variable-length requests will always fit.
No 7B model was loaded or measured on this GPU. The 10 GiB is a KV-only thought experiment; it excludes model weights, activations, workspace, page-table storage, caching allocator reserves and other GPU users. Moreover, parameter count alone does not fix KV size: grouped-query or multi-query attention can use substantially fewer KV heads than query heads, reducing this estimate (Chapter 21). The arithmetic illustrates why demand allocation matters more as per-token KV cost grows; it does not forecast an attainable throughput or batch size.
What the numbers do not show
- Synthetic allocator model, not serving throughput. Lognormal lengths, permanently backlogged admission, one decoded token per resident request per step, no prefill duration and an assumed per-preemption surcharge.
- Utilisation has a specific denominator. The 97–99 per cent figure is valid-token occupancy of allocated/reserved slots during the sampled period, not the fraction of the entire GPU KV pool occupied.
- Declared-limit comparator needs correction. The simulator reserves at most prompt plus 512 tokens but can generate past that allowance; 92/3,000 chat-like and 681/3,000 short-prompt/long-answer seeded requests do so.
- Reservation failure with copy-on-write needs atomicity. Current tests do not cover a partially shared last block followed by an out-of-blocks failure later in the same reservation.
- Gather-based implementation, not a fused paged kernel. The measured penalty is specific to indexed gathers, padded tensors and the hand-written model; production-kernel overhead is unknown.
- GPU measurements are session-specific. Two timing runs are reported, with no hardware traffic counters and no matched request-end-to-end service trace.
- Limited correctness coverage. Four GPT-2 sequences for 20 decode steps, one shared 40-token prefix with two branches, plus small-model CPU regression tests.
- No swapping, real arrivals, fairness, priority-aware preemption, cache hit rates, or production scheduler comparison.
Inference Lab increment
The Lab now has:
BlockAllocatorprovides block IDs and reference counts;BlockTablesmanages sequence mappings, reservations, fork and copy-on-write. Unit tests cover a failed fresh multi-block allocation, but a failed allocation after partial-block copy-on-write needs a rollback regression test.PagedKV,prefill_pagedandforward_paged_decodeimplement a preallocated physical KV pool and a gather-based, masked ragged-batch forward path; they are not a fused PagedAttention kernel.simulate(policy, workload, budget, block_size, watermark)is seeded and deterministic. It returnssteps, extra-penalty step totals, preemption counts, resident concurrency, andmemory_utilisation(logical tokens divided by resident reserved/allocated slots). The declared-limit output cap and the admission/recomputation model need strengthening before the results can be treated as service-capacity guidance.- Reproducible entry points are
python examples/ch14/paged_cache.py simulate,python examples/ch14/paged_cache.py decode, andpython examples/ch14/paged_cache.py gather; the last two require CUDA and an available GPT-2 checkpoint.
The operational rule: report valid KV tokens, allocated block bytes, free pool bytes, mean and peak resident sequences, preemptions, actual recomputation work, and service latency separately. Block-packing efficiency is not pool occupancy, and neither alone predicts throughput. Block tables buy flexible capacity; the kernel and the scheduler must still turn that capacity into correct, affordable execution. Chapter 15 tests the scheduling side of that contract.
What you should now be able to do
- Distinguish advance reservation, final-block slack and allocator-level fragmentation without treating them as the same measurement.
- Build a block table and explain how reference counting and copy-on-write preserve two diverging continuations.
- Compute per-token KV bytes, last-block waste and the difference between allocated-block packing and whole-pool occupancy.
- Explain why a gather-based reference path can be slower than an optimised paged-attention kernel, and what would have to be profiled to account for that difference.
- State how a watermark affects concurrency and preemption in the given simulator, without claiming an observed production throughput benefit.
- Identify an under-reservation bug when generated output exceeds the declared limit, and the atomic-rollback requirement for partial-block copy-on-write.
- Separate demonstrated GPT-2 cache correctness, synthetic scheduler results, and arithmetic projections for larger models.
Key terms
- Physical block/page: a fixed-capacity pool region containing KV data for a number of token positions.
- Logical block table: a per-sequence map from logical positions to physical block IDs.
- Reference count: the number of live block-table owners of a physical block.
- Copy-on-write: making a private block copy when a branch must modify shared data.
- Advance reservation: capacity committed for possible future generated tokens before they exist.
- Internal fragmentation: unused slots within a reserved region or the final allocated KV block.
- External fragmentation: free space split into pieces that cannot satisfy a requested contiguous allocation.
- Block-packing efficiency: valid logical KV token slots divided by resident allocated block slots; distinct from GPU-pool occupancy.
- Preemption: removing a resident request to reclaim its KV blocks, with work that must later be resumed or recomputed.
- Admission watermark: the fraction of pool capacity withheld as free headroom when admitting new prompts.
Reader exercises
- Reproduce the block-size waste table and calculate both resident block-packing efficiency and fraction of the total 32,768-token pool allocated. How can one be 98 per cent while the other is much lower?
- Fix
reserve_declaredby enforcingoutput <= declared_limit, or by growing reservations safely. Regenerate the comparison table and explain which policy rankings change. - Add a test that forks a sequence with a partly filled shared block, then attempts a reservation requiring more blocks than remain free. Make
reserveatomic: failure must restore references, tables and copy queues to their earlier state. - Replace newest-request preemption with a policy based on already-generated work. Measure preemptions and replayed tokens as well as total simulated steps for all three workloads.
- Add a cost function derived from Chapter 9’s prefill and decode timings instead of a constant four-step surcharge. Does the best watermark change when recomputation scales with prefix length?
- Compare the stand-alone indexed gather with a contiguous copy at identical tensor shapes; then profile a complete decode step’s gather, attention and dispatch costs separately.
- Derive the 7B-class projection for 32 versus 8 KV heads, and report the KV-only estimate separately from the model’s full-device memory requirement.
References
- Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al. (SOSP 2023) — primary paper on paged KV blocks, sharing and the vLLM serving system; its reported 2–4× system throughput improvement is not reproduced by this simulator.
- SGLang: Efficient Execution of Structured Language Model Programs, Zheng et al. (NeurIPS 2024) — RadixAttention and cache-aware reuse across related requests; motivates Chapter 16, not measured here.
- Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention, Gao et al. (USENIX ATC 2024) — storage hierarchy and scheduler-aware retention of KV state across turns; not implemented here.
- vLLM source code — implementation reference for production-oriented serving, to be evaluated against a specific version rather than assumed equivalent to this hand-written cache.
Evidence in this chapter
| Claim | Evidence and limitation |
|---|---|
| Fixed-size blocks, block tables, reference counts and copy-on-write | Established in the PagedAttention paper; selected mechanisms implemented and unit-tested here |
| Maximum reservation has 30–61% packing; paged blocks have 97–99% packing | Synthetic simulation; ratios concern resident reserved/allocated slots, not total GPU KV-pool occupancy |
| Last-block waste is approximately \((b-1)/(2\bar L)\) | Analytic approximation under a modulo-uniform assumption; reported comparison differs at large block sizes |
| Ragged-batch GPT-2 paged decode reproduces four independent streams | Book result: four prompts, 20 steps, identical greedy tokens; maximum absolute logit difference 1.129e-3 |
| Forked 40-token prefix shares blocks correctly for two branches | Book result: three blocks shared initially, four after append; one experiment |
| Gather-based decode costs up to 3.3× static at batch 64, context 512 | Book result: two GPU runs; no fused paged-attention kernel tested and no hardware traffic counters |
| Indexed gather is nearly indifferent to physical block order | Book result: one fp16 gather shape; timing range depends modestly on block size |
| Zero-watermark admission causes many preemptions | Book simulation result; strongly affected by fixed extra preemption penalty |
| Five-to-ten-per-cent watermarks reduce preemptions | Book simulation result, but long-answer workload still has 613–1,501 preemptions |
| Declared-limit baseline is a valid capacity constraint | Not established: 92 chat-like and 681 long-answer sampled requests exceed their reserved allowance |
| Reservation rollback is atomic after a partial-block COW | Not established: current code mutates the shared-last-block mapping before later allocation may fail |
| KV-only 7B-class capacity of roughly 20 full-length or 54 mean-length sequences in 10 GiB | Arithmetic based on 32 KV heads and 0.5 MiB/token; not a full-device capacity or performance benchmark |
| Fused paged kernel, real request latency and throughput, swapping, real traces | Not measured |