Tokens, Chat Templates and Special Tokens
Derive a byte tokenizer, inspect pinned chat templates, and construct checked input IDs, attention masks and unshifted assistant labels.
A training example can contain the right question and the right answer while asking the model to learn the wrong thing. The conversation looks sensible in a text editor. Its tensor of labels nevertheless tells the optimiser to reproduce the user’s instructions, ignore the assistant’s answer, or continue beyond a missing end marker. Those mistakes occur before we choose a learning rate or an adapter rank.
Chapter 05, Selecting a Starting Checkpoint, established the identities of four candidate tokenizers and their templates. We now use those verified artifacts to inspect the numerical interface itself. Our destination is a small batch whose token IDs, padding, target positions and causal alignment can all be checked by hand. We will build an educational tokenizer to expose the mechanism, then inspect the real cached tokenizers without loading a language model.
Text becomes an indexed sequence
A Python string is not the direct input to a language model’s numerical layers. An encoder maps text to an ordered sequence of integers. A model with an embedding table uses each integer to select a row of that table. If the embedding width is $d$, the lookup at position $t$ has the form
That is why a tokenizer belongs to a checkpoint’s interface. The vocabulary gives a shared interpretation to IDs used by the input embeddings and the output predictions. A new tokenizer with the same vocabulary size can assign different meanings to every row. Matching tensor dimensions would not make that substitution correct. Our educational tokenizer will make the mapping inspectable; it will never replace a Qwen tokenizer.
Characters, bytes and tokens are different counting units. The character é becomes the two UTF-8 bytes 195 and 169. A tokenizer might represent those bytes with two entries, combine them into one entry, or combine them with neighbouring text. A token can also contain only part of a multibyte character. Decoding individual token pieces may consequently show replacement characters even when decoding the complete sequence reconstructs the text correctly.
Normalisation is another choice. The single character é and the sequence e followed by a combining accent can look alike while having different underlying bytes. Our implementation preserves that difference. It does not lowercase text, remove whitespace or normalise Unicode. These are declared properties of this implementation, rather than assumptions to make about every tokenizer.
Construct a vocabulary from bytes and repeated pairs
Start with a deliberately plain encoder: each UTF-8 byte value is its own ID, from 0 to 255. This representation can encode any valid UTF-8 text without an unknown-word entry. Its weakness is sequence length: an unfamiliar string remains representable, but may require many tokens.
Byte-pair encoding adds entries for repeated adjacent pairs. In our small implementation, count pairs separately within each training string, select the most frequent pair, replace its non-overlapping occurrences from left to right, and repeat. Equal counts are resolved by the lowest integer pair. That tie rule matters: an underspecified tie can create a different vocabulary on another run. The broad pair-merging mechanism and the use of a 256-byte starting alphabet are described in the Hugging Face BPE course; the precise training strings, tie rule and implementation here are our own educational choices.
The executable corpus contains four synthetic strings:
low lower lowest
low low lower
help helpful helper
route routed routing
ByteBPE.train in examples/ftlab/src/ftlab/tokens.py learns at most twelve merges. IDs 256–261 are reserved for our educational control tokens, so learned entries begin at 262. The first three observed merges are:
| Rank | Pair of existing IDs | New ID | Bytes represented |
|---|---|---|---|
| 0 | 108, 111 | 262 | lo |
| 1 | 262, 119 | 263 | low |
| 2 | 32, 263 | 264 | space followed by low |
The third entry is a useful warning against imagining tokens as words. A space can be part of a token. A frequent fragment can combine with its surroundings, and a long word can still split into many pieces. Whitespace is information in the stream; a tokenizer need not represent it as a separate item.
Encoding applies the learned merge rules in rank order to the byte sequence. Decoding concatenates the stored bytes and decodes the complete result as UTF-8. Our tests check exact round trips for whitespace, punctuation, code, unseen text and multibyte characters. They also check that training cannot merge the last byte of one corpus document with the first byte of the next.
This is a simplified byte BPE, not a reproduction of production Qwen tokenisation. It has no production pre-tokenisation rules, no learned normaliser and no claim of compression optimality. With only twelve merges, it is intentionally easy to inspect and poor at compressing many examples. Learning those vocabulary merges is a small corpus-counting exercise; it is not an update to a language model’s weights.
Measure the representation rather than estimating it from characters
The experiment encodes ten formats with the educational tokenizer and with all four verified Qwen tokenizers. These are synthetic examples, not evaluation holdouts. The following are observed lengths from the Docker CPU run recorded in the reference experiment:
| Format | Example feature | Educational tokens | Tokens in each of the four tested Qwen tokenizers |
|---|---|---|---|
| Ordinary text | Route the fictional ticket. |
26 | 5 |
| Whitespace | leading spaces, tab and newline | 23 | 7 |
| Unicode | café Ελληνικά 中文 |
29 | 13 |
| Emoji | Ready 🙂 🚦 |
15 | 5 |
| Punctuation | Wait... really?! (yes/no) |
25 | 8 |
| Code | a short Python function | 35 | 10 |
| JSON | a route and an abstention flag | 36 | 12 |
| Markdown | a heading and list item | 27 | 8 |
| Log text | timestamp, error and numeric code | 46 | 28 |
| Rare string | unretokenisability_739xyz |
25 | 10 |
The rightmost column reports agreement for these particular inputs. It does not establish vocabulary equivalence between the four checkpoints. Their chat formatting and special-token configurations already provide counterexamples to such an inference. Full IDs, raw token pieces and decoded text are retained in the report, so the comparison can be inspected beyond a length total.
There is no reliable fixed conversion from character count to token count. In the unmerged byte exercise, A takes one token and 🙂 takes four. A learned merge can shorten a repeated substring without shortening another string of the same character length. For a sequence budget, use the selected tokenizer on the actual serialised input, including its control tokens and template text.
A conversation is a serialisation protocol
A list of dictionaries is a convenient application representation of a conversation. Its role fields are not automatically separate numerical input channels. A chat template serialises that structure into text and control tokens before tokenisation. The Transformers 5.17.0 chat-template guide explains this conversion and the need to avoid inserting special tokens again when separately encoding an already rendered template.
Our educational template is deliberately different from Qwen:
BOS
USER bytes of the user message EOS
ASSISTANT bytes of the assistant message EOS
It assigns PAD=256, BOS=257, EOS=258, SYSTEM=259, USER=260 and ASSISTANT=261. These integers are local teaching conventions. compile_dialogue inserts the role and end markers explicitly, while ordinary content passes through the byte encoder. A literal string resembling a role name in content remains ordinary bytes; it cannot silently become a role ID in this format.
The compiler supports an optional initial system message followed by alternating user and assistant messages. A completed training dialogue must end with an assistant message. A generation prefix must instead end with a user message, after which compile_prefix appends the assistant role marker. It appends no completed assistant answer or closing EOS. That distinction identifies where generation is supposed to begin.
A real template can insert additional system instructions, newlines, reasoning markers and different message terminators. We do not reconstruct it by concatenating strings that merely look right. inspect_cached first uses Chapter 05’s verify_tokenizer_cache, then loads the exact revision with local_files_only=True and trust_remote_code=False. It also checks the loaded template’s digest against the candidate manifest. None of these operations constructs a model.
For our short fictional prompt and the answer Received., the observed default formatted lengths differ:
| Candidate and mode | Complete sequence | Generation prefix | Final suffix targets |
|---|---|---|---|
| Qwen2.5-0.5B base, default | 31 | 27 | 4 |
| Qwen2.5-0.5B-Instruct, default | 41 | 37 | 4 |
| Qwen3-0.6B-Base, default | 24 | 16 | 8 |
| Qwen3-0.6B, default | 24 | 16 | 8 |
| Either tested Qwen3 candidate, non-thinking | 24 | 20 | 4 |
The Qwen2.5 templates insert different default system material. A base-model label therefore does not imply that the tokenizer has no chat template. Conversely, successfully rendering a template does not show that the base model has learned to follow the resulting conversation. We have not loaded its weights or generated an answer.
The Qwen3 observation is especially instructive. For this completed answer, default and non-thinking mode produce the same completed token sequence. Their generation prefixes differ. Non-thinking mode places the empty thinking block inside the prefix, moving the boundary from 16 to 20 tokens. Under our declared final-suffix objective, that changes which positions receive labels even though the complete text is unchanged. Template arguments belong to the definition of the training example, not merely its display preferences.
Separate message endings, sequence endings and padding
A special token has a declared structural role, but the names of those roles are not interchangeable. BOS marks a beginning where the selected format uses one. An end-of-message marker closes a turn. An EOS configuration tells a generation system about an ending convention, subject to its actual stopping configuration. PAD fills unused batch positions. A checkpoint may reuse one ID for more than one purpose.
The captured tokenizers expose a concrete distinction. In both tested base candidates, EOS and PAD are 151643, while <|im_end|> maps to 151645. In the two non-base candidates, EOS is 151645 and PAD is 151643. All four report no BOS token in this inspection. These are measured tokenizer configurations; we have not tested generation stopping behaviour.
It would therefore be wrong to identify every occurrence of a padding ID as padding, or to assume that every message boundary has the configured EOS ID. The position’s role follows from the serialisation and the batch mask. The same numerical value can appear as an attended ending and as an unattended filler. We will demonstrate this directly after constructing the labels.
Build the smallest batch worth checking by hand
Use two educational conversations. The first contains user Hi. and assistant OK. The second contains user ABCDE and assistant XYZ. Their complete sequences have lengths 10 and 13. We right-pad the shorter sequence to width 13 and convert the arrays to CPU torch.int64 tensors.
For the first row, the report contains:
input_ids: 257 260 72 105 46 258 261 79 75 258 256 256 256
attention_mask: 1 1 1 1 1 1 1 1 1 1 0 0 0
labels: -100 -100 -100 -100 -100 -100 -100 79 75 258 -100 -100 -100
The IDs 72, 105 and 46 are the UTF-8 bytes of Hi.. The IDs 79 and 75 are O and K. The role marker at position 6 establishes the assistant boundary. Our declared objective supervises the assistant’s content and its ending, so positions 7, 8 and 9 carry targets. It does not supervise the role marker, the prompt, BOS or padding.
The complete two-row batch has shape $B \times T = 2 \times 13$. There are 23 attended positions and seven supervised target positions. Those numbers describe different things. The prompt must remain available as conditioning context even though we are not asking the model to reproduce it as an output target.
The accompanying code is small enough to inspect:
sequence = compile_dialogue(tokenizer, messages)
batch = make_batch([sequence, longer_sequence])
validate_batch(batch, [sequence, longer_sequence])
input_ids = torch.tensor([r["input_ids"] for r in batch["rows"]], dtype=torch.long)
attention_mask = torch.tensor([r["attention_mask"] for r in batch["rows"]], dtype=torch.long)
labels = torch.tensor([r["labels"] for r in batch["rows"]], dtype=torch.long)
The validator compares the supplied arrays with freshly compiled expected sequences. It does not accept a self-reported loss mask as independent proof that the labels are correct. Changing a prompt label, dropping an attention position or replacing the assistant ending produces a diagnostic failure.
The figure connects the three mechanisms the section builds separately: merges apply in rank order, the worked row colours prompt, answer and filler positions with their two masks, and the shift panel shows the single alignment from logit positions to target positions. Note how position 9 stays a supervised ending even though its ID doubles as padding elsewhere.
This remains an input-structure check. A caller must supply the intended messages, tokenizer and objective honestly. The validator cannot decide whether an answer is true, whether its source has the necessary rights or whether the dataset is independent of an evaluation holdout.
Attention masks and target masks solve different problems
For each position $t$, let $a_t$ indicate whether it belongs to the real sequence rather than right padding. Let $m_t$ indicate whether that position supplies a supervised target under the selected objective. Our label construction is
The padding attention mask does not itself describe the decoder’s causal restriction. A causal decoder also prevents a position from attending to future positions. With zero-based positions, that restriction permits keys at positions $s \leq t$ and excludes $s>t$, alongside the padding policy. The model implementation combines these constraints; a two-dimensional array of ones is not permission to see future answers.
Likewise, assigning -100 does not remove a token’s embedding from the forward computation. It marks the corresponding target as ignored by the chosen loss convention. Earlier prompt representations can still influence supervised answer predictions and, where parameters are trainable, contribute to their gradients. Loss masking is not a general privacy or information-removal mechanism.
The sentinel -100 is not a vocabulary entry. PyTorch’s cross-entropy interface supports an ignored target index, conventionally -100; its exact semantics are also checked against the installed 2.6.0 implementation. A pipeline using a different loss must honour the intended ignore convention explicitly. This chapter constructs the targets; Chapter 08 derives how the selected targets enter the numerical loss.
An optional second batch in the report sets PAD equal to our educational EOS, 258. At positions 9–12 of the first row the input IDs are then all 258, but their labels are 258, -100, -100, -100. Position 9 is a genuine supervised assistant ending; the following three are fillers. Masking every token whose ID equals PAD would erase a useful ending target. The correct distinction comes from position and attention, not ID equality alone.
Shift exactly once
A decoder predicts the next token from the tokens already available. In the first row, the representation at the assistant role position must predict O. After consuming O, the next prediction is K; after K, it is the ending token.
| Logit position | Target position | Target ID | Interpretation |
|---|---|---|---|
| 6 | 7 | 79 | assistant marker predicts O |
| 7 | 8 | 75 | O predicts K |
| 8 | 9 | 258 | K predicts the assistant ending |
The exported labels are unshifted: a target for input position $t$ is stored at position $t$. When implementing the comparison explicitly, align logits[:, :-1] with labels[:, 1:]. The corresponding shifted target shape in this batch is $2 \times 12$. A model whose forward method already shifts labels internally should receive the unshifted labels instead. Performing both shifts would train a different prediction problem.
The target mask must follow the target position. Although the role-marker position has an ignored label of its own, its logit position predicts the first supervised answer token. Masking logits using the unshifted position mask would incorrectly remove that first answer prediction. causal_pairs records both indices so this distinction can be inspected without guessing from array names.
Truncation changes the question
A maximum sequence length is a limit on the serialised token sequence, including the template. Slicing the string before tokenisation is not equivalent to clipping IDs afterwards. Neither operation automatically preserves a complete conversation or a useful target.
The educational compiler refuses implicit truncation. An explicit whole_turns policy may retain a prefix ending at a complete assistant turn. If the limit cannot retain even one such turn, it reports NO_COMPLETE_TARGET. It will not present a clipped fragment of an answer as a valid shorter example. The report shows a two-turn sequence shortened to its first completed turn and records both original and retained lengths.
The Qwen inspection also executes the tokenizer’s actual right-side truncation at a limit of 16. This is recorded as a diagnostic preview, not an approved training batch. In several inspected modes that limit retains none of the answer suffix. A truncated input can remain structurally encodable while losing the very behaviour we intended to supervise.
Different objectives can justify different policies. A language-modelling exercise may deliberately learn from bounded text windows. A structured answer task may require a complete final object and its ending. A conversation task might remove earlier complete turns. Select that policy before producing labels, and report how many targets survive; do not silently discard examples with no targets and then call the reduced set the original dataset.
Inspect a real template without pretending to have a general collator
The production adapter uses a narrow, explicit objective: supervise the final assistant suffix, including the template’s closing tokens. It renders the preceding conversation with a generation prompt, renders the completed conversation separately, and requires the first to be an exact prefix of the second in both text and IDs. This check matters because matching character prefixes do not universally imply matching token prefixes.
The final labels consist of -100 for the verified prefix and the actual IDs for its suffix. The adapter also pads two completed examples through the real tokenizer and checks the resulting IDs and attention masks before extending labels with ignored positions. It compares tokenisation performed by apply_chat_template with manual encoding of the rendered text using add_special_tokens=False. A mismatch fails rather than selecting a convenient alternative tokenizer or silently repairing the result.
The educational objective supervises every assistant’s content and ending. The production preview supervises only the final suffix. Those are two legitimate declared exercises, not identical objectives. Closing newlines are included in the production suffix, and the thinking-mode observation shows why the boundary must remain explicit.
The adapter rejects unsupported role/content structures and literal tokenizer control strings embedded in message content. This is a conservative input boundary for this exercise, not a universal chat-data rule. Tool messages, multimodal segments, packed examples and general assistant-span discovery need additional adapters and checks. A template’s existence does not supply those semantics for free.
Inspect the batch before training
Use the CPU container, whose runtime network is disabled and source mount is read-only:
docker compose -f docker-compose.book.yml run --rm book-tests python examples/ch07/token_demo.py --output /results/my-ch07-NEW.json --batch-output /results/my-batch-NEW.json
docker compose -f docker-compose.book.yml run --rm book-tests
Choose fresh output paths. The experiment reads the four caches below .local/ch05-tokenizers/<candidate>/<revision>/; it verifies captured file sizes and hashes before use. It neither re-downloads missing files nor substitutes another revision. A clean recipient without those artifacts receives an explicit missing-dependency result for the tokenizer experiment, rather than fabricated IDs.
The educational and mutation tests run without those caches. They check empty and whitespace inputs, unseen UTF-8 bytes, deterministic pair selection, role ordering, generation boundaries, special tokens, padding, label placement, a single causal shift, vocabulary changes and truncation. The package smoke builds and installs FineTuneLab 0.6.0 offline in an ephemeral directory, checking that the educational component imports without PyTorch, Transformers or jsonschema. The tensor example itself uses the separately pinned PyTorch environment.
The following chapters use the unmerged byte/control configuration for a fully transparent small model, while retaining this learned-merge configuration for vocabulary inspection. The worked batch above contains only unchanged original byte/control IDs. Matching those particular arrays does not make a model with 262 output entries compatible with arbitrary learned-merge IDs. The training checkpoint must bind its actual tokenizer configuration.
What this batch establishes
The useful product of this chapter is a declared sequence of integer inputs and an independently inspectable selection of next-token targets. We have executed the educational byte mechanism, four pinned tokenizers, template modes, actual padding and truncation, and a tiny CPU tensor batch. The separate Docker GPU check establishes that small CUDA tensor operations work on the reference machine; it does not establish that any language model or training configuration fits.
No language model has been loaded or evaluated here. No benchmark has been frozen, and no adapter has been trained. The integration requirements concerning scoring authority, operational constraints, fresh resource admission, source binding and weight-content verification remain open before the relevant later experiments. Token inspection adds no training permission to the draft task contract.
Before using a batch, inspect its last prompt token, first answer token, ending, padding and supervised positions. Then establish where the one causal shift occurs. Chapter 08, Next-Token Loss and Teacher Forcing, can consume the exported batch directly and ask the next numerical question: what penalty do the selected next-token predictions receive?
Further Reading
- Rico Sennrich, Barry Haddow and Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units” (2015). Read the source. The paper that brought BPE into sequence modelling, showing merges learned from corpus counts as reimplemented by the chapter’s ByteBPE.
- Hugging Face contributors, “Byte-Pair Encoding tokenization” (living documentation). Read the source. A worked BPE implementation with the same training-then-rank-ordered-encoding structure, useful for checking merge-application order.
- PyTorch contributors, “torch.nn.CrossEntropyLoss” (PyTorch 2.6 documentation). Read the source. The API contract showing ignored targets contribute neither term nor count, which is why -100 must never be confused with padding.