Autoregressive Generation

Generating 64 tokens from GPT-2 on my CPU took 11.5 seconds. I then took the 64 tokens it had produced, appended them to the prompt, and ran the model once over the whole 81-token sequence. That took 0.42 seconds, and at every one of the 64 positions the model’s most probable next token was exactly the token the slow loop had chosen.

Producing the answer was 27 times slower than checking it. That asymmetry is the most important structural fact about generation, and the rest of Part III is largely a sequence of ways to exploit it. We should be careful about the number, and I will qualify it below, because the slow loop is a naive one and a competent loop would narrow the gap. But the direction is not in doubt, and it rests on a distinction that is easy to blur: autoregressive generation is serial in which tokens are chosen, and parallel in how tokens are scored.

This chapter puts the forward pass of Chapter 4 inside a loop, measures what the loop costs, explains the asymmetry, and then does the less glamorous work that decides whether a generation loop is usable in a service: stopping, and streaming text to a client without corrupting it.

Where this sits

Chapter 1 gave the loop in ten lines. Chapter 3 supplied the selection rule and the reason a greedy loop repeats itself. Chapter 4 supplied the function the loop calls. Here the loop becomes an object with a contract: it takes a way of getting scores, a way of decoding tokens, a prompt and a set of limits, and returns text, a list of per-step records, and a reason for stopping.

The Inference Lab piece added is that contract, along with two small components that every serving path needs: a stop-string matcher that works at the level of text, not token ids, and an incremental decoder that never emits half a character. Both are tested without a model.

For the Pi series this is where the earlier agent material meets inference. An agent loop built on text completion commonly cuts the model off at a marker such as Observation:, and a user interface that shows text as it arrives depends on streaming. Both are properties of the loop in this chapter, and both have failure modes that look like model failures and are not.

The loop

Let \(s_t\) be the token sequence after \(t\) steps, starting from the prompt \(s_0\). One iteration is

What the mathematical state of the loop is, and what the operational state is, are different things. Mathematically the state is the token sequence \(s_t\): everything the next distribution depends on. Operationally a runtime may also keep derived data, most importantly the per-layer keys and values of the processed positions, to avoid recomputing them. That cache is an optimisation of the work, and it does not change the distribution the model defines. Chapter 10 returns to the conditions under which that remains true.

What the naive loop costs

The loop in examples/ch05/real_generation.py is deliberately the naive one: at each step it runs the model over the entire sequence so far and uses only the last row of the result. It is the baseline Chapter 10 improves on, and it should be measured before it is improved. The number of positions it processes over \(n\) steps from a prompt of \(P\) tokens is

These are medians of two runs each, so the spread is visible in the results file but not much can be said about it. Two things stand out.

The waste grows without bound. At 128 tokens the loop processed 72 times as many positions as there are distinct tokens. A naive loop is not a slightly inefficient version of a good one, it is quadratic where a good one is linear.

The time did not grow as fast as the position count. Positions rose 63-fold from 8 tokens to 128, and time rose 29-fold. The final column explains it: the cost of processing one position fell from 5.9 ms to 2.8 ms as the inputs got longer. That is the same effect Chapter 4 measured, in which the processor delivered about 31 GFLOP/s on a 16-token input and about 70 on 128, because small matrix products underuse the machine. The practical consequence is that position count is a poor linear predictor of time, and that a cost model needs to know about hardware utilisation as well as arithmetic. Chapter 11 builds that model.

What this table does not give you is the time a cached loop would take, and I am not going to estimate it. That comparison is NOT_RUN until Chapter 10, where the same prompt and lengths are run with a cache.

Latency profile of one generation

Take the 64-token run alone and look at it step by step, discarding a warm-up pass first:

The 95th percentile is more than twice the median, and a reader used to latency numbers from services would take that as variance. It is not. It is almost entirely drift. The median of the second half of the steps is nearly double that of the first half, because each step does more work than the last. If you plot the 64 step times, they are a rising line with a little noise, not a cloud.

This matters for how the Lab reports latency. A percentile computed over a quantity that has a trend answers a question nobody asked. In the naive loop inter-token time is a function of position, so it should be reported by position, or the loop should be fixed so that the trend disappears. With a KV cache the per-step cost becomes nearly independent of position, and percentiles start to mean something again. Chapter 36 takes up what to report and why.

Why checking is cheaper than producing

The experiment from the start of the chapter: generate 64 tokens greedily, then feed prompt plus output as one sequence and take the argmax at each of the 64 positions that precede an output token.

The agreement is the causal mask doing its job, which Chapter 4 showed to be exact. The logits at position \(t\) depend only on tokens up to \(t\), so the logits computed for the full sequence in one pass are the same logits the loop saw one at a time. Generation is a way of computing the same function as the parallel pass, one position later each time.

That is the whole point. The loop is serial because it has to decide \(x_{t+1}\) before it can ask about \(x_{t+2}\). Given the tokens, scoring is parallel. Suppose something else proposes the next several tokens, say a small, cheap model, or a lookup in the prompt. Then the large model can check all of them in a single pass and keep the prefix on which it agrees. That is speculative decoding, and Chapter 17 is its subject. The prompt itself is the same idea turned inwards: all of its positions are known in advance, so they are processed in one pass, which is the prefill of Chapter 9, and the one-token-at-a-time phase that follows is decode.

Be careful about what the 27× shows. It compares a naive no-cache loop with one parallel pass, and the naive loop wastes most of its work. A cached loop would be much faster than 11.5 s and the ratio against the parallel pass correspondingly smaller. I will not guess how much smaller. The correct reading is the qualitative one: scoring a sequence of known tokens costs about one forward pass, not one per token. Second, the agreement is for greedy decoding, where the check is “does the argmax match”. For sampled generation the acceptance rule is more subtle, and Chapter 17 is where it is derived.

Stopping

A generation loop ends for one of four reasons, and each belongs to a different layer.

The first three are inside the loop. The fourth arrives from outside, and a loop that only checks it between steps has a response time of at least one step: with the naive loop at the end of a long generation that is nearly half a second during which a cancelled request is still consuming the machine. Chapter 30 treats this seriously. For now, the loop returns a structured stop_reason (eos, stop_string or max_tokens) because “why did it stop” is a first-class fact in a request record, not something to reconstruct from the output.

How often does a model choose to end?

It is easy to assume that EOS is the normal way for generation to finish. For this model it is not. I ran six prompts for 64 greedy steps each, including a prompt that ends like a finished document and one set up like a chat turn:

<|endoftext|> (id 50256) was never the most probable token in any of the 384 steps. In the document-ending case it reached probability 0.117 and second place, so the model was not indifferent to it, but the greedy rule never selected it. Every run ended because the request said so.

This is a base model from 2019 that was never trained to conclude a reply. A model trained to follow instructions behaves differently, and I have not measured that. The engineering point survives the caveat: a service that treats the token limit as a rare safety net will be wrong for models that rarely emit EOS, and its cost per request is determined by what it allows, not by what the model “means”. Notice also that EOS is not text. The loop stops on the token without emitting it, and whether it appears in the stored sequence is a policy the record should state.

Stop strings

A stop string is a condition on text, such as a marker that ends an agent’s turn. Implementing it on token ids does not work, and the tokeniser shows why:

The same string can be one token or three, depends on what precedes it, and is split at points that have no relation to word boundaries. Even a two-newline stop string has two token forms: it is token 628 when the text tokenises that way, but the model can equally produce token 198 twice, and the two decode to identical text. A stop on the id sequence [628] would silently miss the second.

So the match must be on decoded text. That creates a second problem, because streaming has already sent the text. If the stop string is END and the model produces E, N, D in three tokens, a naive implementation would have emitted E and N to the client before it could know that D would complete the marker. Output the client has seen cannot be retracted.

The solution implemented in StopStrings is to hold back any trailing text that is a proper prefix of a stop string. After E arrives, the matcher holds it; if the next text is X, it releases EX; if it is ND, the stop fires and the held text is discarded along with everything after it. The cost is latency: up to the length of the longest stop string minus one character is delayed. The tests exercise both outcomes, and one in particular: that a stop string completed across several tokens produces text that never contains any part of it.

A matcher that is correct is not a matcher that is cheap to reason about, and it has an invariant worth stating: the text emitted by a stopped generation does not depend on how the stream was chunked. Exercise 2 turns that into a property test.

Streaming

Streaming does not change the computation. It changes when the result leaves the machine, and in doing so it exposes a problem that batch generation never has.

A token is not a character. GPT-2’s tokeniser represents text as bytes, and a character outside the ASCII range may be spread across several tokens, none of which is valid UTF-8 on its own. I decoded token by token the string 推論 café 😀, which is nine tokens:

The naive method corrupts every character that spans tokens: here the two Japanese characters and the emoji. The accented é survived only because this tokeniser happens to have a single token for café. This is easy to miss in testing, because English text tokenises into whole characters, and it is the commonest way for a streaming interface to show garbage in production. The fix in IncrementalDecoder is to decode the whole prefix each time and withhold the output while it ends in the replacement character, which marks an incomplete sequence. It is simple, and it has a cost I should state: it re-decodes the entire sequence on each token, so it is quadratic in length, and it delays emission slightly. A production decoder tracks byte state instead. For a teaching implementation the clarity is worth the cost, and it is the Lab’s reference against which a faster one can be tested.

Two further points follow from this. First, a stream event is not a token. Characters can arrive late because a token was incomplete, held-back text can arrive in a burst when a stop-string prefix resolves, and a transport may batch several deltas into one packet. Measuring inter-token latency from the client’s perspective measures all of that together. Second, the only reliable record of what the model generated is the id sequence, and the only reliable record of what the client saw is the emitted text. The record should keep both.

The loop as code

The Lab’s generate function is about forty lines. The part that matters is the order of operations in the body:

for step in range(max_new_tokens):
    logits = next_logits(ids)                      # the seam: any engine plugs in here
    best = argmax(logits)
    ids.append(best)
    steps.append({"step": step + 1, "token_id": best, "seconds": ..., "positions_processed": len(ids) - 1})
    if best == eos_token_id:
        reason = "eos"; break                      # EOS is checked before it can be emitted
    out, stopped = stopper.feed(decoder.push(best))
    emit(out)
    if stopped:
        reason = "stop_string"; break

Three decisions are encoded there. next_logits is a function from ids to scores, so the loop does not know whether it is calling a toy, a naive transformer, or an engine with a cache; this is the seam Chapter 6 uses. EOS is tested on the token before it reaches the decoder, so it is never emitted. And every step records positions_processed, so a request record can show how much work the loop did, not just how long it took. The toy generation_loop.py from the original chapter, which runs a tiny random transformer with EOS and a token limit, remains as a fast fixture.

What the numbers do not show

• A 2019 base model, greedy only. The EOS behaviour in particular says nothing about instruction-tuned models. Sampling changes both the repetition seen in Chapter 3 and the verification story in this chapter.
• No cache. The waste factors and the 27× are properties of the naive loop. The comparison with a cached loop is NOT_RUN until Chapter 10.
• One prompt for latency, scaling and verification. The per-step drift is a consequence of the algorithm and should be general; its size is not.
• Two repeats per length. The scaling table is indicative.
• Streaming was exercised on the tokeniser and on a synthetic byte-level model, not through a network server. Client-visible latency, back-pressure and disconnects are untouched. Chapter 27 builds that.
• Cancellation and timeouts are described, not measured.
• One CPU thread. Absolute times will differ elsewhere, as will the position-time relationship.

Inference Lab increment

The Lab now has the generation loop as an object with a contract:

• generate(next_logits, decode, prompt_ids, max_new_tokens, eos_token_id, stop_strings, on_text), returning stop_reason, the generated ids, the emitted text, a per-step record and the total time.
• StopStrings, a text-level matcher with hold-back, and IncrementalDecoder, a streaming-safe detokeniser.
• positions_naive and positions_cached, the cost model for the loop.

Together with the earlier chapters a request record can now say: which artefact, which tokeniser, which selection rule and seed, which forward-pass implementation (verified against the oracle), how many positions it processed, how it stopped, and what text was emitted. What it cannot yet do is run fast. Chapter 6 assembles these pieces into the first complete engine, and Part II explains how to make it quicker without changing what it computes.

What you should now be able to do

• Write the generation loop with its dependency structure, and say precisely which part is serial and which is parallel.
• Compute the number of positions a naive loop processes and the waste factor against a cached one.
• Explain why scoring a known sequence takes about one forward pass, and what that implies for speculative decoding and prefill.
• Report stop reasons as a structured field and name the layer that owns each.
• Implement a stop-string matcher that works across token boundaries and does not leak a prefix it later discards.
• Stream text from a byte-level tokeniser without emitting broken characters, and say what the cost of doing so is.
• Explain why percentiles of inter-token latency are misleading in a loop whose step cost depends on position.

Key terms

• Autoregressive generation: choosing tokens one at a time, each conditioned on those before.
• Prefix: the token sequence generated or supplied so far.
• Stop reason: the recorded cause of termination (eos, stop_string, max_tokens, or an external signal).
• EOS token: a vocabulary entry the model can choose to signal that the sequence is complete.
• Stop string: a text pattern at which generation is cut off.
• Hold-back: delaying emission of text that might become part of a stop string.
• Incremental decoder: converts a growing id sequence into text deltas that never end mid-character.
• Waste factor: positions processed by a loop divided by positions that were distinct.
• Time to first token (TTFT): the delay from the request to the first generated token.
• Drift: a systematic change in per-step cost with position.

Reader exercises

1. Add a timeout stop reason checked between steps. Run it with a limit shorter than one step and show that the loop cannot stop in the middle of a forward pass. What does that imply for cancellation?
2. Write a property test: for random splittings of a string into deltas, the output of StopStrings (emitted text plus the stopped flag) is identical regardless of how the string is split.
3. Run the EOS experiment on an instruction-tuned model if you have one cached. How do the stop reasons and the EOS probabilities change?
4. Repeat the parallel-check experiment with sampling at temperature 1.0. Why does the check no longer agree at every position, and what would a correct acceptance rule need to be?
5. Fit the per-step time against position for the 64-token run and compare the slope with the FLOP model from Chapter 4. Is the drift explained by the arithmetic alone?
6. Measure how many characters the hold-back delays for the stop string Observation: on a real generation. Is the added delay noticeable at 4 tokens/s? At 50?
7. Make IncrementalDecoder linear-time by tracking pending bytes. How would you test that it still agrees with the quadratic version?

References

• Attention Is All You Need — the model whose causal masking makes the parallel check possible.
• The Curious Case of Neural Text Degeneration — decoding strategies and repetition under likelihood-maximising loops.
• Transformers generation strategies — official documentation of stopping criteria and decoding options in a widely used implementation.

Evidence in this chapter

Implementation status

• Manuscript: content/books/inference/05-chapter.md
• Runnable examples: examples/ch05/generation_loop.py, examples/ch05/real_generation.py
• Tests: tests/test_ch05_generation.py, tests/test_ch05_real_generation.py (the stop-string matcher, incremental decoder and loop are tested on a synthetic byte-level model; the real-model commands are not exercised by tests)
• Evidence: research/ch05/sources.md, research/ch05/results.json, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about cached generation, instruction-tuned models or network streaming. Chapter 6 assembles tokeniser, engine, sampler, loop and record into the first complete inference engine.