Model Architecture Versus Runtime Architecture

Seven things were changed about how Llama-3.2-1B was run, one at a time, on twelve prompts of 48 greedy tokens each: a chunked prefill (twice), a different attention kernel (twice), a static cache, a padded batch, and float32 arithmetic. In 79 of the 84 runs the output was identical to the baseline’s, token for token. The other five differed at a token where the model had been within 0.02 logits of a tie, three of them an exact tie, and the largest change to the logits at the end of any prompt was 0.027.

Then five things were changed that look similar in a configuration file: rounding the weights to 8 or 4 bits, imposing a window, imposing a window with a few sink tokens, removing the checkpoint’s RoPE scaling. Of 51 runs, 8 gave the same output. The perturbation to the logits ran from 0.41 to 13.5, and the outputs of the larger ones diverged within the first few tokens.

That is the boundary this chapter is about. The earlier chapters of Part 4 each introduced something that changes what a model computes (experts, a head layout, a window, a positional scheme). Chapters 10 to 18 introduced things that were supposed not to. This chapter measures that distinction instead of asserting it, then asks what a runtime should do when a request needs something it does not have. The answer the software stack here gave, over thirteen probes, was to refuse six, to warn and carry on three times, and to say nothing four times, two of which were silent substitutions.

Where this sits

Chapter 2 introduced the manifest, the record of exactly which artefact was run. Chapter 6 gave the engine a backend contract: encode, decode, next_logits and a manifest. This chapter asks what belongs in each, using everything since: a cache (Chapter 10), paging (14), batching (15), prefix reuse (16), speculation (17), graphs (18), expert dispatch with a capacity (19), head layouts (21), windows (22), positions (23). Chapter 25 begins Part 5, turning the loop into a service, and a service needs what this chapter builds: a way to know, before it accepts a request, whether it can honour it.

The Inference Lab piece added here is the architecture fingerprint (what a configuration decides), the admission function (accept, warn or reject before work starts, naming the rule), and the preservation harness (change a knob, compare the function). For the Pi series, the point is what belongs in a provenance record: every option that is not function-preserving is part of the artefact’s identity.

What the model decides

Everything in a checkpoint’s configuration is a decision the model’s training made and a runtime inherits. Reading ten cached configurations into a fingerprint:

None of these has experts. (The window column is what the cached configuration lists. Whether a window applies to every layer or only some is not recorded in these files, so the fingerprint does not claim it; the Gemma-2 architecture is documented to alternate local and global layers, which I did not check against its code here.) Each row fixes things a runtime cannot choose: the size of the cache per token (Chapter 21), the longest sequence the positions can address (Chapter 23), whether some layers see a window (Chapter 22), and, for Gemma-2, an attention logit cap that a fused kernel must honour. A fingerprint is cheap to compute and the first row of any compatibility matrix: it says what the checkpoint demands before anything is loaded.

What the runtime decides, and what it must not

The runtime chooses how the computation is done: which kernel, how the prefill is split, how the cache is laid out, how requests are batched, what precision the arithmetic uses, whether a graph is recorded. The claim of the earlier chapters is that these choices should not change the function. Each chapter checked its own: paged and independent decoding gave the same tokens, with logits within 1.1e-3 (Chapter 14); the schedulers of Chapter 15 returned each request its stand-alone answer, on a small model; a reused prefix matched a full prefill to 7.6e-5 with the same next token (16); recorded and eager decoding gave the same 129 tokens (18); speculative greedy decoding matched plain decoding exactly at \(k = 1, 2\) and, at \(k = 3, 4, 6\), differed on two prompts at tokens that were near-ties (17).

The new measurement puts them in one table, on one model, with one yardstick. The baseline is Llama-3.2-1B in fp16 with the library’s default attention, a one-shot prefill, a dynamic cache and one sequence at a time. For each variant I ran twelve 300-token passages of this book and 48 greedy tokens, recorded whether the output was identical, where it first diverged, what the baseline’s margin was at that step (its top logit minus its second), and how far the logits at the end of the prompt moved.

(The quantisation rounds the weights of the decoder layers and computes in fp16, so it measures the effect of the rounded weights and not of low-precision kernels. The padded batch uses prompts of different lengths and compares each row with its solo run. The baseline’s own margin was within 0.1 at 4.7% of steps, within 0.5 at 25% and had a median of 1.24, so a divergence at 0.5 or 1.0 is not a coin flip.)

Three things stand out.

The runtime changes did move the arithmetic, and rarely moved the output. Every one of them reordered floating-point operations, and the logits at the end of the prompt moved by up to 0.027 for it. Over 576 generated tokens per change that never mattered except where the baseline had two candidates within 0.02 logits of each other: five times in 84 runs. (The 512-token chunk row is a control in effect, since a 300-token prompt fits in one chunk; without it the count is five in 72.) Three of the five were exact ties, all in the batched variant. I did not test why; different shapes plausibly select different kernels, which would decide which of two equal fp16 logits wins. It is the same kind of event Chapter 17’s speculative decoding met: a runtime option is not bitwise-preserving, and a test of one must say what tolerance it is testing to. Float32 arithmetic gave the same tokens in all twelve, so on these prompts the fp16 baseline agreed with the higher-precision run.

The model changes are of a different order. The logit perturbation at the end of the prompt was 15 to 500 times larger than any runtime change’s (0.41 to 13.5, against at most 0.027). With int4 weights, a window or no RoPE scaling, none of the 39 runs matched, and the first divergence came after about 6%, 1%, 9% and 4% of the output on average. Their divergences are at margins up to 1.7, where the baseline was fairly confident.

One of the model changes sits between. Rounding to int8 left 8 of 12 outputs unchanged, and its four divergences were at margins of at most 0.12, which is the range of the runtime flips. By the margin criterion alone it looks like a runtime change; by the size of its perturbation (0.41, fifteen times the runtime changes’) and by the fact that it replaces the weights, it is a different artefact. The boundary is not a line on which every change falls clearly to one side. A deployment that quantises is making a model decision that happens to be gentle, which is why Chapter 12 treated quantisation as something to be evaluated and not assumed.

The control row matters as much as the others. Imposing a window longer than the prompt goes through the same masked-attention path as the shorter window, and changes nothing but the kernel’s route: 12 of 12 identical. So it is the window, not the mechanism for imposing it, that changes the function.

A runtime parameter can be a model parameter

The table says the boundary is not where the code lives. Several things that a serving system exposes as runtime settings change what is computed. Chapter 19’s capacity factor is a runtime setting, and with a finite capacity a token’s output depends on the other tokens in its batch; Chapter 22’s window is a mask applied by the runtime and, on a model not trained with one, a different function; Chapter 23’s RoPE variant is a configuration value and the model’s quality depends on it; truncation of an over-long prompt is a policy and it changes the input. The working criterion is whether the output remains a function of the weights and the request alone, up to numerics. Anything that breaks that, or replaces the weights, is part of the artefact and belongs in the manifest.

What the stack does when asked for something it cannot do

A runtime that cannot honour a request has three honest options: refuse it, warn and say what it did instead, or do something else without saying. I asked the software stack here for thirteen things it could not (or might not) do and recorded which it chose.

Of the thirteen, six were refused, three warned and went on, and four passed silently; of those four, two (Llama’s template, the static cache without compilation) did what was asked and two did something else. The messages were not uniform in quality. The refusals mostly named the cause, but one was an index error with no mention of the context length, and one (the static cache) stopped for a reason, a compilation dependency, that nothing in the request had mentioned: a feature was silently coupled to another. The warnings are the dangerous category: the right-padding warning was accurate and the output of the first row was still wrong, and the “indexing errors” warning for TinyLlama came from the tokenizer, a component that cannot stop a model from being run on the result. A warning in a log is not a decision anyone made.

The silent substitutions are the worst. The Mistral template accepted a system prompt and put it somewhere other than a system turn, which a test that checks for the presence of the system text would pass. The bfloat16 request ran at less than half the speed with no message, which a status code and a plausible output would never reveal. (I timed it; I did not compare its outputs with float16’s.)

Admitting a request before any work is done

The starter chapter’s advice was that unsupported combinations should fail before work is admitted. Here is that advice made into a function. A ModelSpec is the fingerprint above; a Request carries what the caller wants (prompt and generation lengths, sampling settings, system prompt, batch and padding, dtype, attention implementation); RuntimeFacts holds the things about the runtime and hardware that are facts to be checked, not hopes (whether flash attention is installed, the compute capability, how the model’s template treats a system role). admit returns a decision with every reason that applies, each naming the rule and the numbers:

decision = admit(spec, Request(prompt_tokens=1200, max_new_tokens=8, temperature=0.5), facts)
# Decision(admitted=False, reasons=[
#   "context: 1208 tokens exceed the 1024 positions of gpt2",
#   "sampling: temperature is set but do_sample is false, so the library would ignore it"])

Applied to twelve requests that mirror the probes:

Every probe except the static-cache compilation failure is turned into a rejection or a warning before the model is touched. The context rules are a design choice: a context beyond the original training length but inside a shipped extension is a model decision, so it is admitted with a warning, whereas a context beyond a model’s positions with no extension is rejected, because Chapter 23 measured what TinyLlama’s loss does there (it triples).

Two honest limits. The rules are as good as their facts. system_role is a fact about a chat template, learned by rendering it; the compute capability comes from the device; whether flash attention is installed comes from an import. If a fact is wrong the function is wrong, so the facts want a conformance test of their own (render the template, run the kernel) and not a table typed by a person. The rules came from failures. The static-cache compilation failure is not covered by any rule above; I wrote the rules after seeing the failures they cover, and that failure only appeared when the preservation run crashed. A capability list is a record of what has been tested, and it has the gaps of what has not.

Reading a release as a matrix

Compatibility is a matrix of rows, and “loads the weights” is one. Two first-party releases make the point from opposite directions, and I report only what their READMEs say, as the authors’ statements.

DeepSeek’s DeepInfer README, at the commit pinned in the book, lists one supported model (DeepSeek-V4.1-Flash) and two accelerator families (NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs), names asynchronous scheduling among its features, and says batch-invariant inference and deterministic sampling are NVIDIA-only. It describes the release as offline inference with the core features needed for serving at scale, and lists as required for a complete service but not included a KV cache system, prefill/decode disaggregation and load balancing across replicas. A runtime that supports one model on two families of hardware, and says which rows differ between them, is a precise matrix; the rows it lists as absent are part of it.

OpenAI’s gpt-oss README says its models were trained with the harmony response format and “should only be used with this format; otherwise, they will not work correctly”, and that a caller using generate directly must apply the format through the chat template or its library. It also says its PyTorch, Triton and Metal implementations are reference implementations and not meant for production. I did not run these models (they are not in the local cache), so the claim that output is wrong without the format is the authors’. For the matrix, loading the weights is one row, the response format another, and a reference implementation shows a third: that the model can be run at all.

PagedAttention (Kwon et al., SOSP 2023) is a runtime contribution: its abstract reports near-zero waste of KV cache memory and 2 to 4 times the throughput of the systems it compares with, at the same latency. The abstract says nothing about model output; the claim is about the runtime, with the model held fixed. Chapter 14 checked that standard on its own implementation (identical tokens, logits within 1.1e-3), which is not a check of the paper’s.

What the numbers do not show

• One model. The preservation table is Llama-3.2-1B in fp16, twelve prompts of book text. Larger models, other families and other prompts may flip ties at different rates.
• Greedy decoding. Sampling hides ties differently (Chapter 17’s distribution test is the check for it) and was not part of this table.
• The quantisation is simulated. It rounds weights and computes in fp16. Real low-precision kernels add their own arithmetic.
• Twelve prompts per row, three for RoPE. The margins of the divergences are real data points; the rates are not precise.
• One software stack. The refuse/warn/silent classification describes PyTorch 2.6.0 with transformers 5.17.0 on this GPU, as the library behaves now. A different release, or llama.cpp, vLLM or SGLang, would fill the table differently, and I ran none of them.
• The admission rules encode what I tested. They are not a complete list.

Inference Lab increment

The Lab now has:

• fingerprint and ModelSpec: the architecture facts of a configuration, with the derived cache size, attention kind and whether the RoPE is an extension.
• Request, RuntimeFacts, admit and Decision: admission by named rules, with every reason reported.
• greedy, summarise_knob, first_divergence, near_tie_share: the preservation harness, with margins at divergences.
• fake_quantise_linears: weight rounding on a grid, for experiments.
• fingerprints, preservation, fallbacks and admission commands.

The rule this adds: every runtime option is either function-preserving, with a test that names its tolerance, or it produces a different artefact and belongs in the manifest; and a runtime refuses what it cannot do before it admits the work, naming the rule. Capacity limits, windows, positional schemes, truncation and (gently) quantisation are the options that look like settings and are not.

What you should now be able to do

• Read a configuration into a fingerprint and say what it fixes for the runtime.
• Test a runtime option for function preservation: choose a baseline, compare tokens and logits, and interpret each divergence by the baseline’s margin.
• Explain why a runtime change can flip an output without having changed the function, and why a model change is recognised by the size of its perturbation and the earliness of its divergence.
• Classify a request failure as refused, warned or silent, and say why silent substitution is the worst.
• Write an admission function whose rules name their numbers, and say what each rule’s facts must be tested against.
• Read a vendor release as a matrix of supported rows, and say what its stated gaps imply.

Key terms

• Architecture fingerprint: the facts a configuration decides (attention layout, cache per token, positions, window, positional scheme).
• Function-preserving: a change after which outputs are the same up to numerics.
• Margin: the top logit minus the second at a decoding step.
• Near-tie: a step whose margin is within rounding of zero.
• Admission: the decision, before any work, to accept, warn about or reject a request.
• Silent substitution: doing something other than what was asked, without saying so.
• Capability matrix: the set of tested rows of what a runtime supports for a model.
• Conformance test: a test that verifies a declared capability.

Reader exercises

1. Repeat the preservation table on Llama-3.2-3B. Do the runtime changes still flip only ties?
2. Add speculative decoding and CUDA graphs (Chapters 17 and 18) as rows. Where do they fall?
3. Add rules to admit for the static-cache compilation dependency, and for a system role in a template that folds it.
4. Replace the typed RuntimeFacts for a model by a function that derives them: render the template, check the import, read the device.
5. Run the same probes on another software stack and compare the refuse/warn/silent counts.
6. Make a quantised model’s perturbation visible earlier: what is the smallest number of bits at which the outputs of the twelve prompts begin to differ?
7. Add the capacity factor of Chapter 19 as a knob on a mixture model. Which side of the boundary does the preservation test put it?
8. Write the manifest entry for a deployment that uses a window, int8 weights and a RoPE extension. Which fields change the artefact’s identity?

References

• openai/gpt-oss — first-party model repository; harmony format requirement and reference-implementation caveats; read on 22 October 2026.
• deepseek-ai/DeepInfer at 774ce244 — pinned first-party repository; supported model and hardware, included and not-included components; read on 22 October 2026.
• Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., SOSP 2023 per the arXiv page; read on 22 October 2026.

Evidence in this chapter

Implementation status

• Manuscript: content/books/inference/24-chapter.md
• Runnable examples: examples/ch24/adapters.py (original toy adapters), examples/ch24/contract_lab.py (fingerprint, admission, preservation and fallback experiments)
• Tests: tests/test_ch24_adapters.py, tests/test_ch24_contract_lab.py (fingerprints of GQA, MHA, windowed and extended-RoPE configs, every cached config parses, each admission rule and the reporting of all reasons together, the divergence summary and near-tie share, the weight-rounding grid and its error bound; CPU only)
• Evidence: research/ch24/sources.md, research/ch24/results.json, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about any other runtime, about models above 1B parameters in the preservation table, or about the correctness of any vendor’s production system. Chapter 25 begins Part 5 with the inference loop and the API around it, where the admission function becomes part of a service’s front door.