From Training to Execution

The GPT-2 file on my disk is 548,105,171 bytes. The model it contains has 124,439,808 parameters, and each is a 32-bit float, so the parameters occupy 497,759,232 bytes. About 50 megabytes of the file are not parameters at all.

That gap is a small thing, and it is the whole subject of this chapter in miniature. We say “the model” as though it were one object, a block of learned numbers. What actually arrives from training is a bundle of things: tensors that were learned, tensors that were not, a description of how to wire them together, a tokeniser, default generation settings, and the code that interprets all of them. An inference system has to take that bundle and turn it into something that executes. Between the two lie decisions that nobody made explicitly, and several of them change what the model does.

Chapter 1 timed a request and found the forward pass dominated it. It treated the model as given and recorded one number for loading it. This chapter opens that box. By the end you should be able to say what an artefact consists of and how to identify it exactly, which independent switches turn a trained model into an inference model, what each of them costs or saves in measured terms, and why the first request to a process is slower than the second.

Where this sits

Chapter 1’s request record had a load_model stage and a free-form metadata dictionary that held the model name and nothing more. Two defects were left visible on purpose: the record did not capture which revision of the model it ran, and it treated loading as a single event. This chapter fixes the first and breaks up the second.

The Inference Lab piece added here is the artefact manifest: a precise, hashable description of what was loaded, plus a readiness check that reports whether the loaded model is in an execution state. The manifest is what lets later chapters say “this result came from this artefact” rather than “from GPT-2”. That matters for every comparison in the book. Two runtimes that produce the same text from different artefacts, or different text from the same one, tell us different things, and only a manifest can say which case we are in.

For the wider Pi series, the connection is practical. Retrieval, memory and agent volumes call models by name through providers, and a name does not identify an artefact. The manifest is the local, checkable version of “which model was that, exactly?”, and the same question returns, in a weaker form, when we reach provider APIs in Part VI.

What training leaves behind

Training produces parameters. Everything else needed to use them comes from somewhere else. For the model used in this book, the Hugging Face snapshot contains these files:

Only the first file is “the weights”. The rest decide what the weights mean. Change config.json to say 11 layers and the tensors no longer fit. Change the merge rules and every token id shifts, so the same text is a different input. Change generation_config.json and a library call that never mentions sampling will sample differently. None of these changes touches a single learned number, and each changes the system’s behaviour.

The hashes come from examples/ch02/execution_modes.py manifest, and they are a book result: they identify the files that every GPT-2 measurement in these chapters used, snapshot 607a30d783dfa663caf39e06633721c8d4cfcd7e.

The 50 megabytes

The tensor file holds 160 tensors, not 148. The extra twelve are named h.N.attn.bias, one per layer, each of shape 1 × 1 × 1024 × 1024 in 32-bit float. That is the causal mask of the attention mechanism, precomputed and stored as a buffer: state the module needs but training does not update. Twelve of them at 4,194,304 bytes each come to 50,331,648 bytes. Adding the 14,291-byte header that safetensors places at the front of the file gives 548,090,880 + 14,291 = 548,105,171, the exact file size. I checked this by reading the header directly, not by assuming it.

Three consequences matter later. File size is therefore not a measure of parameter count. A mask stored at maximum context length is allocated whether or not any request is that long, which is a small preview of the memory behaviour that Chapter 10 and Chapter 14 are about. And an artefact manifest that lists only parameters would silently omit state that the module nonetheless carries.

Frozen is a policy, not a property

Inference assumes the parameters do not change. It is tempting to think of that as a fact about the model. It is not. A PyTorch module does not know whether it is being trained or executed. Every one of GPT-2’s 148 parameter tensors reports requires_grad = True immediately after loading, in evaluation mode, inside an inference entry point. The same object that served a request a moment ago can be handed an optimiser and trained.

The experiment that shows this is small. Run a forward pass with autograd enabled, take the logits at the last position, and call backward(). It succeeds, and gradients appear on the parameters. Run the same forward pass inside torch.no_grad() or torch.inference_mode() and backward() raises an error: the output has no recorded history, so there is nothing to differentiate. The model is identical in all three cases. What differs is whether the framework was told to remember how the output was computed.

So “the weights are frozen” is a statement about what the calling code does. It is enforced by the way the forward pass is invoked, and an inference system has to establish it deliberately and, ideally, check it. This is also the first place where the book’s evidence discipline applies to a system property. Chapter 1 recorded the claim “no gradients were recorded” for the real model. The check that makes it a result and not an assumption is the readiness check below.

Three switches that look like one

There are three separate things that people loosely call “inference mode”.

1. Module mode, set by model.eval() or model.train(). It changes the behaviour of layers that behave differently in training, notably dropout and batch normalisation.
2. Global autograd mode, set by torch.no_grad() or torch.inference_mode(). It decides whether operations record a computation graph.
3. Parameter trainability, the requires_grad flag on each parameter. It decides whether a parameter can accumulate a gradient at all.

They are independent. Setting one says nothing about the others, and the measurements below show each of the three combinations that matter.

Module mode: dropout

During training, dropout zeroes each activation with probability \(p\) and rescales the survivors so the expected value is unchanged. For an activation \(x\):

Run the same 10-token input through the model twice, with the module set to training mode, and compare the logits. The largest absolute difference between the two outputs was 41.68. Repeat in evaluation mode and the difference was exactly 0. Logits for this model are of order tens, so the training-mode run is not a small perturbation: it is two different computations of the “same” forward pass. In this single case the argmax at the last position still agreed between the two runs, so greedy decoding would have produced the same token. I would not generalise from that. It is one prompt, and a slightly different margin between the top two logits would flip it.

The practical lesson is that a model left in training mode does not fail loudly. It produces fluent output that varies between runs, and the variation looks like a sampling effect or a numerical quirk. Chapter 39 comes back to reproducibility at length, and this is one of the cheapest causes to rule out.

One finding qualifies the usual advice. In Transformers 5.17.0, from_pretrained already returns a module with training = False. Calling model.eval() afterwards is a guard against a library default changing, not the repair of a present defect. The readiness check reports training_mode: false both before and after the call.

Global autograd mode: memory

Leaving autograd enabled during a forward pass makes the framework keep every intermediate tensor needed to compute derivatives later. For inference, those tensors are never used. The cost is measurable.

I ran a single 512-token forward pass under each of the three modes, each in a separate process after a warm-up, and read the process’s peak working set immediately after the forward pass. The peak before the forward pass, reached during model loading, was about 1,240 MB in every process.

With autograd enabled the process peak rose by about 528 MB above its loading peak. Under the other two it rose by about 32 MB. Because peak working set is a process-wide maximum that already included model loading, these numbers are lower bounds on the extra memory each mode needed, but the ordering is not in doubt.

A rough estimate shows why the order of magnitude is plausible. For 512 tokens, the attention score matrix of one layer is 12 heads × 512 × 512 × 4 bytes, about 12.6 MB, and a forward pass with autograd keeps several tensors of that size per layer, plus feed-forward activations, across all twelve layers. The logits themselves, 512 × 50,257 × 4 bytes, are about 103 MB and exist in every mode. This is arithmetic from the model’s dimensions, not a measurement of what the framework saved. It also predicts something testable: the autograd overhead should grow roughly with the square of sequence length, and exercise 4 asks you to check that.

Disabling the graph is therefore not a micro-optimisation. In a long generation without a KV cache, where Chapter 1 showed every step re-running the whole sequence, graph retention would be repeated at every step.

no_grad versus inference_mode

Memory does not distinguish them: 1,272.0 MB against 1,272.5 MB is noise. The difference lies in what the output is allowed to do afterwards. PyTorch documents inference mode as the stricter variant: besides not recording a graph, it skips view tracking and version-counter bookkeeping, and tensors created inside it are marked as inference tensors that cannot later participate in autograd. In the experiment, only the output of inference_mode reported is_inference() == True.

That restriction is the price and the benefit. If any code downstream might need to differentiate through an output, such as a reward model or a gradient-based explanation tool, inference_mode will break it and no_grad will not. If nothing will, inference_mode is the clearer statement of intent. I did not measure a speed difference between the two in isolation. The probe times in the results file include a failed backward attempt and a single warm-up, so they are not suitable for that comparison, and I make no claim about it.

Parameter trainability

Neither eval() nor inference_mode changes the requires_grad flag. All 148 tensors remain trainable after both. This does no harm inside inference_mode, where no graph is recorded, but it means an accidental call outside the context can build a graph over all 124 million parameters. Setting requires_grad_(False) on an inference-only model removes that possibility. It costs nothing to do, and it is the one switch of the three that the experiments here did not need to demonstrate the difference for. I include it as a recommendation, not a measured finding.

The readiness check

The three switches can be inspected. inference_readiness(module) in examples/ch02/execution_modes.py reports the module mode, the number of trainable tensors and the global autograd state at the point of the call, and sets ready to true only when the module is in evaluation mode and autograd is off:

def inference_readiness(module) -> dict:
    parameters = list(module.parameters())
    trainable = sum(1 for p in parameters if p.requires_grad)
    return {
        "training_mode": bool(module.training),
        "trainable_tensors": trainable,
        "total_tensors": len(parameters),
        "autograd_enabled": bool(torch.is_grad_enabled()),
        "ready": (not module.training) and (not torch.is_grad_enabled()),
    }

Run against GPT-2 it reports, after loading, ready: false (autograd is on); after model.eval(), still ready: false; inside torch.inference_mode(), ready: true, with all 148 tensors still trainable. The check does not claim the model is safe. It makes the state visible, so a request record can carry it instead of a hard-coded assumption.

That point deserves emphasis because the original Chapter 2 example blurred it. Its toy path recorded gradients_enabled_during_generation = False and model_training_mode = False as constants, because the toy has no autograd. They were true by construction but not measured, and the toy result lists training_mode among its simulated stages for that reason. The real-model path measured them. A reader comparing the two records would see identical fields with very different evidential status. The simulated_stages list from Chapter 1 exists to prevent exactly that confusion.

Identifying the artefact

If the model is a bundle, we need a way to say precisely which bundle. The manifest has two layers.

File layer. The size and SHA-256 of every file in the bundle, as in the table above. This detects any change to any file. The local cache stores each large file under a name that equals its SHA-256, and for model.safetensors that name is 248dfc3911869ec4…, identical to the hash I computed independently. That agreement is a useful cross-check on the cache layout, though it describes this cache and is not a general property of Hugging Face tooling.

Tensor layer. For each parameter, its name, shape and dtype, summed to 148 tensors, 124,439,808 elements, 497,759,232 bytes, all float32. On top of that, parameter_digest hashes the names, shapes and raw bytes of the parameters in sorted order. For GPT-2 it gives 95cb2531cd09fbf7…. The tensor-layer digest answers a different question from the file hash: whether two loaded models have identical parameters, regardless of how they were serialised. A model converted to another container format, or loaded from a differently ordered file, has a different file hash and, if the conversion was lossless, the same parameter digest. Chapter 12 will need this distinction when a quantised model has the same name and different numbers.

What a hash does not give you is equally important. It shows that bytes are unchanged since they were hashed. It does not show who produced them, that the weights belong with this tokeniser, or that the artefact is the one a vendor claims. Safetensors improves one thing: it is a non-executable container, unlike a pickled checkpoint, which can run code when loaded. But a safe container is not a provenance check. For closed provider models there is no artefact to hash at all, and the manifest idea degrades to recording the model identifier and date the API returned. Chapter 32 treats what a provider response does and does not let you conclude.

The toy analogue

The toy path in examples/ch02/reproducible_generation.py implements the same idea at small scale. A Checkpoint holds the transition table, bias, vocabulary and seed. Saving it writes a 93,259-byte JSON file with its SHA-256, f51790a1c6eb34f8…, and loading recomputes the hash and refuses on mismatch. I perturbed one table entry by \(10^{-9}\) and loading failed. The check works at that granularity because it hashes the canonical JSON, with sorted keys and fixed separators, so two semantically equal checkpoints have equal hashes.

That is deliberately a small analogue. It validates that this file has not changed. It does not validate a format, check version compatibility, or establish where the file came from.

What the first request pays

Chapter 1 treated loading as one number: 1.58 to 3.20 seconds depending on the process. Five fresh processes, each doing imports, a load and four forward passes over the same 7-token prompt, split that number into three costs that differ in kind.

The import cost dominates by an order of magnitude and has nothing to do with the model. It is the cost of starting a Python process with a large framework on this machine, with a cold interpreter and no warm file cache. It is the biggest single reason why “time to first response” in a naively started process looks terrible, and it is also an argument for running a long-lived server instead of spawning a process per request, which is the starting point of Part V.

The model load of about 1.5 seconds is also paid once per process. Resident memory rose from 577 MB after import to 1,089 MB after load, an increase of about 512 MB, a little more than the 497.8 MB of parameters, as you would expect from the parameters plus framework allocation. In every one of the five processes the first forward pass was slower than at least the following three. The pooled median of calls two to four was about 103 ms, against 154 ms for the first, a penalty of roughly 50 ms or 50 per cent. I read that as one-off initialisation work inside the framework, such as memory allocator growth or kernel selection, but I did not isolate the cause. The consistency across processes is the result. The cause is a hypothesis.

This revises something Chapter 1 left uncertain. There I reported one process with a slow first forward step and a repeat run in which the first request showed no clear penalty, and I declined to generalise. The two views are compatible. The penalty is about 50 ms, which is large relative to a single 100 ms forward pass and invisible relative to a request of sixteen of them. Whether a warm-up cost matters depends on the ratio of the penalty to the total request, which is why Chapter 36 reports time to first token separately from throughput.

The numbers are also a reminder that benchmarks need a stated treatment of warm-up. A harness that includes the first call measures something different from one that discards it, and neither is wrong, but mixing them is.

What the numbers do not show

• One machine, one session, one thread. Absolute times do not transfer. The import cost in particular depends on the disk, the antivirus configuration and the state of the file cache.
• Peak working set is a blunt instrument. It is a process-wide maximum on Windows. It captured the autograd overhead clearly because that overhead exceeded the loading peak. A smaller effect could be hidden below it.
• Dropout was run on one prompt. The 41.68 maximum difference and the agreeing argmax are single observations.
• No determinism claim across hardware. Eval mode makes two runs on the same machine agree exactly. It says nothing about whether another CPU, another thread count or a GPU gives the same bits. Chapter 39 measures that.
• No GPU. CUDA is available on this machine and none of these measurements used it. The first recorded Chapter 2 result labelled its device cuda because torch.cuda.is_available() was true, although the weights were never moved and the run executed on the CPU. The record was wrong, and I only found it while reconciling that file with the new runs. The label now reads the device of the model’s parameters, and the result file carries a note. It is a small instance of the larger point: a field that records what was available will be mistaken for what was used.
• The manifest proves integrity, not origin.

Inference Lab increment

The Lab now has two additions to the request record from Chapter 1:

• file_manifest, parameter_manifest and parameter_digest, which together describe an artefact as hashes and tensor inventory.
• inference_readiness, which reports module mode, trainable tensors and autograd state.

A request record should carry the artefact’s file hashes and parameter digest, and the readiness report taken at the moment of generation, not at load time. The state when the model was loaded and the state when it ran are different facts, and the second is the one that determines the output. The next chapter extends the identity story to the tokeniser, the one component of the bundle whose change alters a request without touching any weight.

What you should now be able to do

• List the components of a model artefact beyond its parameters, and say which of them can change behaviour without touching a weight.
• Explain why file size and parameter count differ, with an example from a real file.
• Distinguish module mode, global autograd mode and parameter trainability, and say which of them each of eval(), no_grad() and inference_mode() changes.
• Compute a manifest and a parameter digest for a model and say what each does and does not prove.
• Split “model load time” into import, load and first-call warm-up, and state which of them a long-lived server amortises.

Key terms

• Artefact: the complete set of files and code needed to execute a model, not only its weights.
• Parameter: a learned tensor updated by training.
• Buffer: a tensor a module carries that training does not update, such as a causal mask.
• Frozen weights: parameters that the execution path does not change. A property of the calling code, not of the model.
• Computation graph: the record of operations that automatic differentiation needs.
• Evaluation mode: module behaviour intended for inference, selected by eval().
• Inference tensor: a tensor produced inside torch.inference_mode(), which cannot later be used in autograd.
• Manifest: a hashable description of an artefact’s files and tensors.
• Cold start: the first execution in a process, which pays one-off costs.

Reader exercises

1. Corrupt one byte of the toy checkpoint JSON and confirm that loading fails before generation. Then change the whitespace only, without changing any value, and explain why the result differs from the corruption case.
2. Run execution_modes.py dropout on a model that has no dropout. What does the training-mode run show, and what does that tell you about checking for the problem?
3. Add requires_grad_(False) to the loading code and rerun readiness. Then call the model outside inference_mode. What changed about the output and the memory behaviour?
4. Run execution_modes.py modes with 128, 256 and 512 tokens. Does the autograd overhead grow roughly with the square of the sequence length, as the attention-matrix estimate predicts? What would a linear trend suggest?
5. Convert the checkpoint to another container format, if you can, and compare file hashes and parameter digests. Which one changes?
6. Take the first forward pass out of the timing in your own harness. State what the remaining measurement now describes.

References

• PyTorch: inference_mode — official documentation of inference mode and inference tensors.
• PyTorch: Module.eval — official documentation of evaluation mode.
• Safetensors documentation — official specification of the non-executable tensor container, including the length-prefixed JSON header read directly above.

Evidence in this chapter

Implementation status

• Manuscript: content/books/inference/02-chapter.md
• Runnable examples: examples/ch02/reproducible_generation.py, examples/ch02/execution_modes.py
• Tests: tests/test_ch02_reproducible.py, tests/test_ch02_execution_modes.py (the real-model commands are not exercised by tests; the helpers are, on a tiny module)
• Evidence: research/ch02/sources.md, research/ch02/results.json, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about GPU behaviour, cross-hardware determinism or production checkpoint validation. Chapter 3 turns to the one component of the bundle that every request passes through before any tensor exists: the tokeniser.