What Is Inference?

I asked a 124-million-parameter model to continue a sentence by sixteen tokens. On a CPU it took about 2.7 seconds. The answer was:

An inference engine takes a trained model and a request, and it must turn them into a model that can be used to generate a model that can be used to generate

Three things are true of that result at once. It was slow for something that produced a few dozen characters. It was repetitive to the point of being useless. And it was entirely correct, in the sense that the program did exactly what greedy decoding of that model is defined to do. Run it again and you get the same sixteen tokens.

Most of this book is about holding those three observations apart. Slow, poor and correct are claims about different things, and the habit of attributing a symptom to the wrong layer is the commonest way to waste a week of optimisation. Slow is a property of the computation and the machine. Poor is a property of the parameters and the decoding rule. Correct is a property of the program relative to its specification. An engineer who treats them as one thing will tune the sampler to fix latency, or buy a faster GPU to fix repetition.

This chapter builds the vocabulary and the first measurement. By the end you should be able to say, for one request to one model, which stages ran, which of them were real computation and which were simulated, where the measured time went, and what the measurement does not let you conclude.

Where this sits

This book is one volume in a series that builds a coding-agent system called Pi. The earlier volumes cover the agent loop, language, embeddings, retrieval and memory, and the series plan places optimised language-model programs (DSPy) immediately before this one. Each of them depends on calls to a language model, and in each the cost, delay and behaviour of those calls are a dependency rather than the subject.

This volume opens that box. In the series plan it contributes efficient model execution and provider-level measurements: how a trained model is executed, what that execution costs, and how to measure it honestly. It consumes the calls the earlier volumes make and returns two things: an explanation of what a call does, and an instrument that can tell whether a change to the system made it better. The volumes that follow depend on it: GPU computing takes the kernels apart, model serving puts local models behind provider-compatible interfaces, and evaluation and performance engineering need the measurement discipline built here.

Throughout the book we build one small system, the Inference Lab: an inspectable, pluggable runtime for reproducible experiments. It is not a clone of any production engine and it does not claim to match one. Every chapter adds one piece and says which. The piece this chapter adds is the smallest one: a record of a request and the time each stage took. Chapters 41 to 45 assemble these pieces into a runtime and compare it with established engines.

Four meanings of one word

The word inference is used for four different things, and the confusion between them is the source of most muddled discussion.

1. Statistical inference is the classical idea of drawing a conclusion from evidence under uncertainty. A language model’s output is not a calibrated statistical conclusion. It is a token sequence selected from a distribution, and whether the distribution is well calibrated is a separate empirical question.
2. A forward pass is the evaluation of a neural network on an input. For a transformer it applies embedding, attention, feed-forward, normalisation and output-projection operations and produces scores for the next token.
3. Token generation is the repeated application of forward passes, with each selected token appended to the input of the next. This is autoregressive generation.
4. Production serving is the system that accepts requests, schedules them, runs generation, streams or batches results, enforces limits and reports measurements.

When this book says inference without qualification it means the path from an already trained artefact to an observable result: meanings 2 to 4. Training, architecture research, prompting and product design are outside it. That boundary is not tidy. A prompt changes the tokens and so the work; a quantised checkpoint changes the arithmetic; an architecture decision determines what can be cached. Where those interact with execution, the interaction belongs here. What stays outside is the question of whether the model should say what it said.

The chain that every chapter preserves

Every chapter in this book is an elaboration of one chain:

model parameters -> request -> tokens -> computation -> generated tokens -> served response -> measured outcome

Each arrow is a transformation that can fail independently, and success at one stage does not imply the next. The table lists, for each stage, what it owns, a failure that is easy to mistake for a failure elsewhere, and where the first chapters of the book let you observe it.

    flowchart LR
    P[parameters] --> R[request]
    R --> T[tokens]
    T --> C[computation]
    C --> G[generated tokens]
    G --> S[served response]
    S --> M[measured outcome]
    G -->|append and repeat| C
  

The loop from generated tokens back to computation is the single most important structural fact in the chapter. It is why a response of two hundred tokens costs two hundred forward passes and not one, and it is the reason caching, batching and speculation, which fill most of the later parts of the book, exist at all.

The autoregressive factorisation

A causal language model defines a conditional distribution over the next token given everything before it:

For a batch of \(B\) sequences of length \(L\), a vocabulary of \(V\) tokens, one forward pass produces logits of shape

First, generation only needs the final position’s logits, but the naive implementation computes all \(L\) of them. For GPT-2 the output projection maps a hidden size of 768 to a vocabulary of 50,257, which is roughly 77 million floating-point operations per position, a large share of the per-position cost. Computing that projection for every position and discarding all but the last is waste. How much, and how real engines avoid it, is a topic for the prefill and decode work in Part II.

Second, each step’s input is the previous step’s input plus one token. A naive loop therefore recomputes everything it computed the step before. Whether that is acceptable depends on how expensive the recomputation is, which is the question we measure below and answer properly in Chapter 10.

Written as code, the whole of generation is a loop of about ten lines:

ids = encode(prompt)
for _ in range(max_new_tokens):
    logits = model(ids)[-1]          # forward pass, last position only used
    next_id = select(logits)         # greedy, sampled, constrained...
    ids.append(next_id)
    if stop(next_id, ids):
        break
return decode(ids)

Everything else in this book is either making model(ids) cheaper, making the loop less wasteful, deciding which requests share the machine while it runs, or measuring whether any of that worked.

The Inference Lab, stage by stage

The first implementation deliberately contains no neural network. It exposes the stages a real system has so that each can be timed and, later, replaced. The stages are:

1. request_parse validates and normalises the request.
2. tokenize splits text into model-facing symbols.
3. encode maps symbols to integer ids.
4. forward_pass computes scores for the next token.
5. sample selects a token.
6. decode maps ids back to text.
7. response_format assembles the response.

A production system adds batching, scheduling, memory allocation, caching, streaming and telemetry between and around these. The seven above are the smallest set that still shows the shape.

Each request produces a GenerationResult, defined in examples/ch01/inference_stages.py:

@dataclass
class GenerationResult:
    prompt: str
    tokens: List[str]
    token_ids: List[int]
    text: str
    timings: List[StageTiming]
    simulated_stages: List[str]
    metadata: dict

Two fields matter more than they first appear. timings is an ordered list of (stage, seconds) pairs, so a per-token stage appears once per generated token and the order of events is preserved. simulated_stages is a list of the stages that did not do real work. A result that cannot say which of its stages were simulated cannot safely be compared with one that can.

The ToyGenerator implements the interface with a seeded table. Its forward pass looks up a row of scores for the previous token and applies a numerically stable softmax:

logits = self._logits(previous_id)
maximum = max(logits)
exps = [math.exp(value - maximum) for value in logits]
total = sum(exps)
probs = [value / total for value in exps]

Subtracting the maximum before exponentiating changes nothing mathematically and prevents overflow, a pattern that recurs in attention. Selection is greedy with a deterministic tie-break on the smallest token id, so a fixed prompt, seed and length always give the same result.

Run it from the repository root:

python examples/ch01/inference_stages.py "Hello" --max-new-tokens 8
python examples/ch01/inference_stages.py "Hello" --max-new-tokens 8 --json

For seed 7 the eight generated characters are RWPFdyxf. That output is meaningless as language, and that is the point. The toy is a test fixture for the interface and the timing harness. It is not evidence about model quality, hardware or serving.

What is simulated

Three stages are simulated in the toy path, and the result says so:

• forward_pass is a table lookup and a softmax over 57 symbols: no parameters learned from data, no attention, no matrix multiplication of meaningful size.
• sampling_distribution is greedy selection from that table, not a distribution over natural language.
• kv_cache does not exist. Each step looks only at the previous token, so there is nothing to cache.

The simulated stages are not an apology. They define the contract a later, real implementation must meet. When Chapter 6 assembles the first complete engine around a real backend, the interface, the tests and the timing harness stay the same, and the list of simulated stages gets shorter.

The real model

The same interface drives a real model. RealModelRunner wraps a Hugging Face causal language model, loads it once, and records the same stage timings. It decodes greedily and, deliberately, has no cache: every step runs the model over the whole sequence so far. That is the baseline that later chapters improve on, and it should be measured before it is improved.

Experiment design

The question is narrow: for a small model on a CPU with no optimisation, where does the time of one request go?

• Hypothesis (book hypothesis, not yet a result). Almost all request time is in the forward pass, and the per-step forward time rises as the sequence grows, because each step recomputes the whole prefix.
• Held constant. Model gpt2 (124,439,808 parameters), snapshot 607a30d783dfa663caf39e06633721c8d4cfcd7e, greedy decoding, one prompt of 17 tokens, 16 new tokens, CPU, PyTorch 2.6.0+cu124, Transformers 5.17.0, one PyTorch intra-op thread, offline model cache, Windows 11, a single machine.
• Varied. Number of requests: one process with one request, and one process with five consecutive requests against a loaded model.
• What would falsify it. If non-forward stages accounted for more than a small fraction of time, or if per-step forward time were flat across steps, the hypothesis would be wrong.

python examples/ch01/inference_stages.py "An inference engine takes a trained model and a request, and it must turn them into" --max-new-tokens 16 --real --json
python examples/ch01/inference_stages.py "An inference engine takes a trained model and a request, and it must turn them into" --max-new-tokens 16 --real --repeat 5 --json

Both runs used the model from a local cache. They download nothing and call no paid service. Results are recorded in research/ch01/results.json.

Where the time went

Over five requests (80 forward passes) against one loaded model:

The forward pass accounts for 99.7% of the measured request time. The hypothesis survives its first test, though the test was weak: on a CPU with a tiny vocabulary and no network, a forward pass of a 124-million-parameter model was always going to dominate string handling. The useful information is the size of the gap. Sampling, which is the stage most discussions fixate on, cost less than half a millisecond per token against a forward pass of nearly 180 ms.

Model loading is not in that table, because it happens once per process. It took 3.20 s in the single-request run, 1.58 s in the repeat run, and 2.37 s in the run recorded when this example was first written. Those three numbers come from three processes on one machine. A factor of two between them is a warning that load time depends on file caching and operating-system state, and that a single load measurement is a weak basis for any claim.

The five requests produced identical text, and their durations were 2.70, 2.67, 2.86, 2.82 and 2.92 seconds. There is no sign here of a first-request penalty after loading. That does not mean none exists. In the separate single-request run the first forward pass took 0.403 s against 0.132 s for the second, which suggests one-off warm-up work inside the process. That is one observation from one run and I do not generalise it. The repeat run had the opportunity to show the same thing at request level and did not.

Per-step cost without a cache

The single-request run recorded each forward pass in order. The 16 values, in milliseconds, were:

403.1  132.2  136.8  149.8  157.1  151.3  153.8  164.6
171.4  179.8  179.9  183.4  173.6  188.5  207.6  209.0

Ignoring the first step, which includes the warm-up discussed above, the step time rises from 132 ms to 209 ms, a factor of about 1.6, while the number of positions processed rises from 18 to 32, a factor of about 1.8. The trend is consistent with the hypothesis that each step pays for the whole prefix. It does not prove it. Step position and sequence length are the same variable in this experiment, thermal and scheduling noise sit on top, and I ran it once. A convincing version varies prompt length independently of step number and repeats each point, which is the design of the fourth exercise at the end of this chapter and of the prefill and decode work in Chapter 9.

A rough estimate shows why the order of magnitude is plausible. A transformer forward pass costs roughly two floating-point operations per parameter per token, so GPT-2 needs about 0.25 GFLOP per position. Eighteen positions are about 4.5 GFLOP, and at 132 ms that is roughly 34 GFLOP/s on a single thread. That is arithmetic from published parameter counts, not a measurement of the hardware, and it is accurate only as an order of magnitude. It is useful because it gives a number to compare against once the same computation moves to a GPU and, in Chapter 11, to a bandwidth limit.

The output is a separate question

The generated text above repeats itself. Greedy decoding picks the most probable token at each step, and small models are well known to fall into loops under that rule. Nothing in the engine is broken. The engine was asked for the argmax and returned it, faithfully and identically on every run.

This is the clearest example in the chapter of the stage separation. A reader who sees that output and concludes the inference engine is faulty has attributed a decoding-rule property to the computation. A reader who concludes the model is poor has attributed it to the parameters. The accurate statement is narrower: greedy decoding of this model on this prompt loops. Changing the selection rule would alter the text without touching a single forward pass. Changing the engine would alter the speed without touching a single token.

Toy and real side by side

The toy forward pass had a median of 5.7 microseconds over 80 steps. The real one had a median of 177.9 ms. That is a ratio of roughly 31,000, and it must not be read as an optimisation figure in either direction. The toy does a table lookup. The real model does matrix multiplications with 124 million parameters. The comparison is there to make the boundary visible: the toy exercises the interface and the timing harness, the real model exercises the computation, and only the second is evidence about transformer inference.

Tokens are the unit of work

The toy generator treats each character as a token. Real models do not. A tokeniser maps text to a sequence of integer ids drawn from a fixed vocabulary, and everything that is counted later, including context limits, per-token prices, cache size and throughput, is counted in those tokens rather than in characters or words.

examples/ch01/token_counts.py compares characters, UTF-8 bytes and GPT-2 tokens for four short samples:

The same amount of nominal content costs between 11 and 37 tokens here. The Japanese sentence has fewer characters than the English one and needs more than three times as many tokens. Those are four hand-picked strings and one tokeniser from 2019, so they illustrate the effect and do not estimate it. A tokeniser trained on different data will give different ratios. The consequence is nevertheless general: a request’s cost and delay are functions of its token count under a specific tokeniser, so the tokeniser is part of the model’s identity. Swapping it changes the work without changing a weight. Chapter 3 treats this properly.

What the numbers do not show

The measurements above are narrow, and it is worth being explicit about how.

• One machine, one session. Nothing here is a portable performance result. The environment is recorded so the experiment can be repeated, not so the numbers can be compared with anyone else’s.
• One thread. PyTorch was using one intra-op thread. A configuration using the machine’s 24 logical CPUs would give different absolute times and perhaps a different split.
• One prompt and one model. The per-step trend might look different at 1,000 tokens, and a larger model would shift the balance between forward pass and everything else.
• Wall-clock timing with Python overhead. The timings list measures elapsed time around each stage, including interpreter and library overhead. It does not separate arithmetic from memory movement. Chapter 11 does that.
• No GPU, no batching, no concurrency, no memory measurement. These are not measured. They are recorded as not measured in the results file, not omitted from it.
• Model identity was recorded by hand. The program stores the model name gpt2 but not the snapshot revision. The revision above was read from the cache directory afterwards. Provenance that depends on a human remembering to write it down will eventually be wrong. Closing that gap is the first exercise on the Inference Lab.

Inference Lab increment

This chapter adds the request record: a GenerationResult carrying output, ordered stage timings, the list of simulated stages and metadata. Its interface is the TokenGenerator protocol:

class TokenGenerator(Protocol):
    def generate(self, prompt: str, max_new_tokens: int) -> GenerationResult: ...

Anything that satisfies the protocol can be timed by the same harness, repeated by run_repeated, and summarised by summarise_timings. The toy and the real model already satisfy it, which is the first small demonstration that an engine can be swapped under a fixed interface and measured the same way. The multi-backend chapter, number 43, generalises this idea, and the benchmark chapters rely on it.

Two limits of the record are already visible and are intentionally left in place. It has no notion of concurrent requests, so it cannot yet express queueing delay. And its metadata is a free-form dictionary, so nothing enforces that two results are comparable. Both are addressed in the service and benchmarking parts of the book.

What you should now be able to do

• Say which of the four meanings of inference a given statement is about, and name the meaning that this book uses.
• Explain why generating \(n\) tokens takes \(n\) forward passes, and where a naive loop repeats work.
• Run one request against a real model, read the stage timings, and say which stage dominates and by what margin.
• Distinguish a slow result, a poor result and a correct result, and say which part of the chain each belongs to.
• State at least three things a given measurement does not allow you to conclude.

Key terms

• Inference: executing a trained model on an input to produce an output.
• Forward pass: one evaluation of the model’s layers on a given input.
• Autoregressive generation: producing tokens one at a time, feeding each output back as input.
• Logits: unnormalised scores over the vocabulary before selection.
• Token: the discrete unit a model reads and writes, defined by a specific tokeniser.
• Greedy decoding: selecting the highest-scoring token at every step.
• Sampling: selecting a token randomly according to a distribution derived from the logits.
• Stage: a named step in the request path whose duration is recorded separately.
• Serving: the system that manages requests, execution and responses around the model.
• Simulated stage: a stage that returns plausible output without performing the real computation.

Reader exercises

1. Modify ToyGenerator to sample from the softmax distribution using a fixed seed instead of greedy selection. Which stages change? Which stay identical? Does repeatability survive?
2. Add a stop reason to GenerationResult (length reached, end-of-sequence, stop string). Decide whether the stop reason belongs to generation or to serving, and defend the choice.
3. Extend RealModelRunner to record the snapshot revision and tokeniser name in metadata automatically, and add a test that fails if either is missing. Why is a free-form dictionary a risky place for this?
4. Run the repeat experiment with prompts of 8, 64 and 256 tokens and the same 16 new tokens. Plot median forward time against position processed. Does the shape support the hypothesis that each step pays for the whole prefix? What would it have to look like to falsify it?
5. Set torch.set_num_threads to 1, 4 and 12 and repeat the experiment. Does the forward-pass share of request time change? Does the step-time trend?
6. Write one claim from this chapter that your experiments could falsify, and the measurement that would do it.

References

• Attention Is All You Need — the paper that defines the transformer and causal self-attention.
• Transformers generation strategies — official documentation for decoding rules, stopping criteria and cache configuration around the model.
• PyTorch 2.6 documentation — tensor and module reference, including intra-op thread control.
• deepseek-ai/DeepInfer at 774ce244 — a pinned first-party repository showing how scheduling, kernels, sampling and distributed execution surround the model at a larger scale. Its own documentation says parts of the production service are not in the public release, so it supports a claim about the systems layer, not about any hosted service.

Evidence in this chapter

Implementation status

• Manuscript: content/books/inference/01-chapter.md
• Runnable examples: examples/ch01/inference_stages.py, examples/ch01/token_counts.py
• Tests: tests/test_ch01_inference.py (13 tests; the real-model path is not exercised by them)
• Evidence: research/ch01/sources.md, research/ch01/results.json, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft and draft = false remains set. It makes no claim about GPU behaviour or production readiness. Chapter 2 turns to the artefact this chapter treated as given: how training produces the parameters that inference consumes, and what must be preserved when they are loaded.