← Distillation From First Principles

Distilling Transformers and Instruction Models

Moving distillation from classifiers to decoder-only transformers: tokenisation, causal masking, completion-only loss, token-level versus sequence-level supervision, and a controlled E3 baseline with exactly matched update budgets.

Three groups of transformer students in this chapter’s experiment, three seeds each, reached 0.525, 0.528, and 0.525 mean exact match. The first learned from ground-truth labels, the second from a teacher’s soft token distributions, and the third from the teacher’s own generated outputs. The third matched the first to the third decimal on every seed, and not by coincidence: the teacher was perfect, so its outputs were the labels. A distillation result means little until it is set beside the plain supervised baseline and the reason for any tie is understood.

Chapter 10 distilled sequences with GRUs. This chapter changes the machinery: the teacher and student are decoder-only transformers trained on instructions rather than classifiers trained on labels. The question is:

What changes when the teacher and student are transformers trained on instructions rather than classifiers trained on labels?

The honest answer has two halves. Much changes in implementation: tokenisation, causal masking, completion-only loss, label shifting, vocabulary alignment, template compatibility. Little changes in principle: the same teacher-student logic, the same agreement-versus- correctness discipline, the same matched budgets. And the baseline result is deliberately unglamorous: on a simple task with a perfect teacher, token-level distillation ties supervised fine-tuning and sequence-level distillation is supervised fine-tuning under another name. That null is the foundation Chapters 18 and 19 build on — the interesting distillation regime needs a teacher that can be wrong.

From labels to instructions

The E3 task is unchanged (reverse-complement over symbols {3,4,5,6}, lengths 4–8, deterministic oracle); only its presentation changes. Each example is now a token row:

[BOS, INST_RC, s1..sk, EOS, t1..tk, EOS, PAD...]

The prefix [BOS, INST_RC, symbols, EOS] is the instruction; the suffix [targets, EOS] is the completion. The single instruction token INST_RC names the transformation the way a natural-language verb would — “reverse-complement” compressed to one id — which is exactly the abstraction Chapter 18 needs: swapping INST_RC for trace-step tokens later reuses the whole pipeline. Nine vocabulary ids suffice: PAD 0, BOS 1, EOS 2, four symbols, one instruction token, one spare. Rows are 24 tokens long; splits (1,200/400/1,000 per seed) reuse the disjoint-pair construction from Chapter 10’s data work, with fingerprints recorded per seed.

The single instruction token INST_RC (introduced above) is exactly the abstraction Chapter 18 needs: swapping it for trace-step tokens later reuses the whole pipeline. Nothing about the task got harder. What got harder is the contract between data and model: every token after the prompt boundary is supervised, every token before it is not, and the model must never learn from padding. Getting that contract right is most of this chapter.

The tiny teacher and student

Both models are decoder-only transformers with learned absolute positions, pre-norm blocks, causal attention, and an unmasked next-token head: the teacher is width-32, 2 layers, 4 heads (26,816 parameters); the student is width-16, 1 layer, 4 heads (3,984 parameters). Decoder-only is a deliberate task fit, not a fashion choice: the E3 row is prompt-plus-completion, and a decoder models exactly the conditional distribution of completion given prompt.

Pre-norm and GELU follow standard small-transformer practice; dropout is 0.0 everywhere — determinism first, regularisation nowhere, because a 4k-parameter student on 1,200 examples needs signal, not noise, and every run must reproduce bit-for-bit from its seed. No pretrained weights, no external checkpoint — the contract’s open-weight inventory (Qwen3, DeepSeek distillations) is explicitly not authorised by this protocol, and the study uses neither. Small is a feature here: every gradient, mask, and shift can be inspected, and the full baseline trains in minutes on CPU.

Tokenizer and template compatibility

Logit matching requires identical output spaces. The study’s tokenizer (e3-instr-tokenizer-v1) and template (e3-instr-template-v1) are explicit records: vocabulary size 9, specials {PAD:0, BOS:1, EOS:2, INST_RC:7}, fixed row layout, single-turn prompt-completion template used identically in training and evaluation. Before any distillation run, check_tokenizer_compatible compares the student’s spec against the teacher’s; any mismatch in version, size, specials, or template is a refusal, never a silent projection. A worked refusal: vocab 99 against vocab 9 fails on size; template other-v9 fails on template; an empty specials map fails on specials — three errors, each named. This is the narrow, checkable version of a real production hazard: chat-template mismatch between training and evaluation, or a projected vocabulary that pretends two tokenizers are one. The policy is simple — distil across tokenizers only with a stated, verified alignment, otherwise stop — and the code enforces it where the prose cannot.

A mismatch the check catches

Consider an evaluation harness that “helpfully” wraps every prompt with a header token before decoding: [BOS, HDR, INST_RC, ...] instead of [BOS, INST_RC, ...]. Everything still runs — same vocabulary, same model, no error. But every absolute position embedding shifts by one, so the trained position-to-role mapping (position 1 means “instruction verb”) now points at the wrong token, and greedy outputs degrade without a single exception being raised.

This is the chat-template mismatch of the contract’s failure list in miniature: training template v1, evaluation template something-else, silent behaviour change. The compatibility check exists to make this class of error loud: the eval harness must present the same template version, or the run refuses before a single misleading number is recorded. The fixture in the test suite does exactly this with other-v9 — a two-line demonstration that a version string, checked, is worth more than a paragraph of assumed consistency.

Causal masking, made explicit

A decoder-only model predicts each token from the tokens to its left, and the implementation must guarantee that — not merely encourage it. The causal mask is an upper-triangular boolean matrix: position \(i\) may attend to positions \(j \le i\) and to nothing else. For three tokens the allowed pattern is:

. X X
. . X
. . .

(rows query, columns key, X masked). The model adds learned absolute positions to token embeddings before the first block, then applies pre-norm attention-plus-MLP blocks with that mask. Two details matter. First, padding never participates: prompts are bucketed by length for evaluation so no PAD enters a decoded row, and training rows are fixed length 24 with the completion mask excluding PADs from the loss (the encoder still reads PADs, exactly as the GRU precursor did — a stated, shared imperfection, not a silent one).

Second, the mask is rebuilt per length and cached, and a sequence longer than max_length is refused rather than truncated mid-attention. Tests cover mask shape, causality (no upper-triangular leakage), and the length-24 ceiling. Attention costs \(O(L^2)\); at \(L=24\) that is noise, and the chapter says so instead of performing scaling analysis on a toy.

Masking: completion-only loss

The loss sees the completion and nothing else. For the row [1, 7, 3, 4, 2, 6, 5, 2, 0, 0], the completion mask is [F, F, F, F, F, T, T, T, F, F]: positions after the first EOS up to the padding. The boundary rule is strict — the first EOS terminates the prompt; a missing boundary or a boundary before position 3 is an error, not a guess.

Padding is never supervised: the mask stops at the first PAD, and an all-masked row raises instead of returning a silent zero (which would read as perfect loss). The classic failure this prevents is training on padding tokens: with PAD id 0 also a valid target index in an unmasked loss, the model spends capacity predicting padding and the metric quietly rewards it. Completion masking is not an optimisation; it is the definition of the task.

Label shifting, done right

Next-token training pairs decoder inputs row[:-1] with gold outputs row[1:]. The off-by-one failure is worth demonstrating because it is silent: shift the wrong way (inputs as targets) and the model learns to copy the prompt — loss falls, exact match stays at zero, and nothing in the loss curve confesses. The worked check on [1, 7, 3, 4, 2, 6, 5, 2, 0, 0] pins the split: inputs end [..., 5, 2, 0], gold ends [..., 2, 0, 0] — gold position 2 holds 4, the next symbol, where the input holds 3.

A test asserts exactly this alignment. Shifting is a one-line operation with a chapter-level blast radius; the chapter treats it accordingly. So is its quiet companion: the completion mask must shift with the gold. The mask is computed on full rows (positions of the completion), then sliced as mask[:, 1:] alongside gold = rows[:, 1:] — mask the pre-shift positions instead and the loss supervises the prompt’s last token while ignoring the final EOS, a two-token rotation of the entire objective that still converges to something plausible-looking. The runner applies both slices in one place, and a test asserts the masked positions equal the known completion span on a fixture row. Shift the labels, shift the mask, or supervise the wrong thing beautifully.

One row, three contracts: layout, mask, shift

Figure 1. The worked row [1, 7, 3, 4, 2, 6, 5, 2, 0, 0]: prompt grey, completion blue, PAD light; the mask admits T positions only; the shift pairs each decoder input with the next gold token (gold position 2 holds 4 where the input holds 3). Worked row, not a measurement.

Packing, truncation, and context limits

Rows are 24 tokens long; the models accept at most 24 (max_length). Anything longer is refused with an explicit error, never silently cut — silent truncation moves the EOS, which moves the supervision, which moves the metric, all without a log entry. This study does not pack multiple examples into one row: with a fixed 24-token canvas and examples up to 20 tokens, packing would save little and risk much.

The chapter states the boundary-safe packing rule it does not need, so the reader recognises it when they do: pack only whole prompt-plus-completion units, mask the seams (a completion must never attend to a previous example’s completion as context — causal masking alone does not prevent that, because seam tokens are legitimate left context), and record the packing map in provenance. Truncation, where a task genuinely exceeds context, belongs to the TaskSpec as a declared policy (truncate-where, refuse-when), not to an ad-hoc slice. Here the policy is total: lengths 4–8 always fit, anything else is a specification error caught before training.

Instruction-following checks

Exact match is not the whole contract. An instruction follower must also: terminate (emit EOS rather than rambling to the step limit), stay in vocabulary (no out-of-range ids), and keep the response inside the completion (not echo the prompt). The audit measures all three: invalid-output rate 0.000 in every arm on every seed — every greedy decoding terminated with EOS inside the vocabulary.

That perfect score deserves the same scepticism as any perfect score: on this tiny task the output space is 4 symbols plus EOS and the model learns termination within the first updates, so the check is nearly vacuous here. It is included because the harness must exist before the task that needs it — Chapter 18’s traces can ramble, and the same three checks will bite there. An instruction-following check that cannot fail teaches nothing; one that is never run teaches less.

Token-level KL

With aligned vocabularies, the token objective is the Chapter 05 form restricted to completion positions. For teacher distribution \(p_T\) and student \(p_S\) at temperature \(T\):

\[ L_{\mathrm{tok}} = T^2 \, D_{\mathrm{KL}}(p_T^{(T)} \parallel p_S^{(T)}), \]

averaged over masked positions, with teacher tensors detached — no gradient flows into the teacher, verified by test. The mixed arm uses \((1-\lambda)\mathrm{CE} + \lambda T^2\mathrm{KL}\) with \(\lambda = 0.5\), \(T = 2\): temperature softens both distributions before the KL (at \(T=2\) a logit gap of 2 counts as 1), and the \(T^2\) factor keeps the KL gradient on the same scale as the CE gradient — the Chapter 05 derivation applied verbatim, restricted to completion positions.

A worked calculation fixes the scale: uniform student against a peaked teacher (logit 10 on one of 9 ids) gives KL ≈ 2.193 — nearly the maximum \(\log 9 ≈ 2.197\) — while identical distributions give exactly 0. Temperature, mixture weight, and masking are frozen before training; none is tuned against the test set.

The sequence-level objective

Sequence-level supervision trains the student on whole teacher outputs as hard labels: decode the teacher greedily on each training prompt, rebuild full rows, and train with completion-masked cross-entropy. This is Kim and Rush’s sequence-KD idea at minimum scale — greedy decoding, not beam search (the mode is what greedy finds; beam search would find a better mode at higher cost, and the chapter prices that choice by not paying it), no interpolation, just the teacher’s mode as supervision.

Where the teacher is uncertain, greedy decoding keeps the single most likely completion and discards the rest of the distribution — sequence supervision is therefore sharper than token supervision and more brittle: one wrong greedy choice poisons the whole row, while token-KL preserves the uncertainty. On a perfect teacher the distinction vanishes, which is why this chapter cannot separate them and says so. Its honest limitation is visible before running: if the teacher is perfect, its outputs equal the oracle and the “distillation” arm is supervised fine-tuning with extra steps. That is exactly what happened (see below), and the chapter reports it as a property of the regime rather than a bug in the code.

The frozen E3 baseline

Three arms on one shared population per dataset seed (0, 1, 2):

Arm Supervision
A-hard ground-truth oracle completions (SFT reference)
B-token-kl mixed CE + token KL vs the local teacher
C-seq teacher greedy-decoded targets as hard labels

Teacher: 1,200 updates, batch 64, Adam 0.002. Students: EXACTLY 600 updates each, batch 32, same optimiser, same init across arms within a seed, best-validation checkpoint restored after the full budget — the Chapter 15 lesson applied from the start, with a test asserting the manifest’s update counts. The test ledger records the held-out evaluation as the only test-set examination in this stage; no task, threshold, or architecture decision follows from it. Rights: RM-109 (train_and_evaluate_e3_transformer, verified-permitted), gate passed before training. A calibration gate on the teacher guards arms B and C; the next section describes it.

Teacher training and the calibration gate

The teacher trains on oracle rows: 1,200 updates, batch 64, Adam 0.002 — the same optimiser family as every student, deliberately, so that teacher quality reflects the task rather than the tuning. It reached validation exact 1.0 on all three seeds. The calibration gate (proceed only above 0.5) therefore passed trivially, and its trivial passage is worth one paragraph: a gate that never fires still belongs in the protocol, because the arms that consume the teacher (B and C) are uninterpretable without it.

Had the teacher stalled at 0.3, arms B and C would have been blocked honestly — distilling from a teacher that cannot do the task measures nothing about distillation. The gate converts “the teacher might be bad” from an uncontrolled confound into a predeclared stop condition. Here it did not stop anything, which is itself the finding that explains everything below: a perfect teacher leaves token-KL nothing to add and sequence supervision nothing to differ on.

Measured results

Frozen run (E3T-20261115T182053Z-5f416d): 3 teachers + 9 students, held-out greedy exact vs oracle:

Seed A-hard B-token-kl C-seq
0 0.426 0.446 0.426
1 0.564 0.528 0.564
2 0.584 0.611 0.584

Aggregates: A 0.525 ± 0.086, B 0.528 ± 0.083, C 0.525 ± 0.086. Token accuracy ~0.92, NLL ~0.21, invalid rate 0.000 everywhere, all 600 updates each. Three findings, all descriptive. First, token-KL ties SFT (B−A: +0.020/−0.036/+0.027) — soft targets added nothing measurable here, which is consistent with a teacher that is certain and correct but does not show that they never help. Second, C equals A to the third decimal on every seed: the perfect teacher’s outputs are the oracle labels, so sequence distillation degenerates to SFT, as predicted.

Third, student/teacher agreement equals student correctness wherever the teacher is oracle-correct — agreement is a diagnostic, not an achievement, and the chapter refuses the easy story that “high agreement means successful transfer” when both simply match the oracle. Almost: seed 2, arm A, agreement 0.581 against exact 0.584. The 0.003 gap is the whole lesson in miniature — a validation-perfect teacher still errs on some held-out rows, so agreement and correctness can diverge even here.

Where they coincide, agreement carries zero information beyond correctness; where they differ, the difference (not the headline rate) is the datum. The audit reports both numbers separately for every arm and seed, and the manuscript declines to average them into one. The length-band breakdown (seed 0, arm A) shows where the remaining errors live: 0.902 at length 4, 0.803 at 5, 0.420 at 6, 0.355 at 7, 0.228 at 8. Per-token error compounds with length — exactly the difficulty proxy Chapter 13 declared without validating, now observed in a student. The negative is the point: distillation needs headroom — a teacher that can be wrong, a task the student cannot already solve — and the next two chapters construct exactly that regime.

Cost and shape of the runs, for the record. Teachers trained 1,200 updates to validation exact 1.0 on all seeds; students ran exactly 600 updates each (batch 32, ~19,200 presentations, all 1,200 uniques seen — the small space saturates exposure, unlike Chapter 15). Whole study: 438.6 s wall clock on one CPU thread (peak process 533 MB, a whole-process high-water mark, not model-only memory).

Per-arm student training: ~17 s (A), ~20 s (B/C — KL and teacher-decoding overhead visible but small). Student latency ~5.6 ms per single-row greedy decode; 3,984 parameters; invalid rate 0.000. Teacher generation for arm C (greedy over 1,200 training prompts per seed) cost seconds, not minutes — sequence supervision is cheap when the teacher is tiny and the task is short, a ratio that inverts at language-model scale and is recorded here so the inversion is legible later.

Instruction tuning is not distillation

Arm A is supervised fine-tuning on ground truth. Calling it distillation would be a category error the chapter states plainly: distillation requires teacher-derived supervision (KD-010); SFT on oracle labels has no teacher in the loop. The arm exists as the reference every distillation claim must beat, not as a distillation method.

The per-seed B−A gaps (+0.020/−0.036/+0.027, mean +0.004) show what “must beat” means in practice: token-KL neither clears nor falls short of SFT by any margin three seeds can resolve. Conflating the two is how weak distillation results get reported as successes — “the distilled student reaches 0.525” is true and meaningless without “so does plain SFT”. Every comparison in this part keeps the SFT reference visible for exactly this reason.

What changes with transformers

Against the GRU precursor: attention replaces recurrence (parallel training, quadratic context cost, position must be injected, not inherited); the vocabulary is explicit and versioned rather than implicit; masking is structural (causal + completion + padding) rather than incidental. For context — not a claim, the setups differ — the GRU student of Chapter 15 reached 0.42 exact on noisier, smaller data while this transformer student reaches 0.525 on clean full data with fewer parameters (3,984 vs 5,127). That is consistent with data and budget, rather than architecture, being the binding constraint on this task, but the comparison is not like-for-like and does not test it. The same pathologies recur at new addresses — off-by-one shifts, padding supervision, template mismatch. What does not change: agreement versus correctness, matched budgets, frozen splits, rights before training. The chapter’s contribution is portability of discipline, not novelty of architecture.

Limits and the handoff

One synthetic instruction task, three seeds, tiny models, single-task completion supervision — no chat, no multi-turn, no pretraining. Two absences are disclosed rather than filled. First, calibration: token accuracy and NLL are reported, ECE is not computed in this study’s evaluation (the precursor measured it; the transformer runner records the raw logits needed to add it, but the manuscript claims only what it measured). Second, catastrophic forgetting: single-task fine-tuning cannot exhibit it, so the chapter does not pretend to test it — the failure mode is named in the contract because multi-task instruction tuning will need it, not because this experiment measures it. The mechanism is worth one sentence now so it is recognised later: completion-only loss on task B leaves task-A prompt representations free to drift, and nothing in the objective protects them.

Nor is there out-of-distribution evaluation in the strong sense: every length band 4–8 appears in training, so the length decay above is interpolation under compound error, not extrapolation. A single template, a single instruction token, zero dropout (determinism chosen over regularisation at this scale): each is a declared narrowing of the claim, not a hidden one. The teacher’s perfection is both the result’s explanation and its boundary: nothing here says what happens with a fallible teacher, and Chapter 18 is built to find out, using this chapter’s architecture, tokenizer, and update-matching discipline on an arithmetic reasoning task where the teacher’s traces can be wrong in checkable ways. Gate 17 is therefore satisfiable on the evidence above: implementation tested, losses and masking verified, records frozen, rights passed, results reconciled, manuscript hash-recorded.

References

Sanh, V., Debut, L., Chaumond, J. and Wolf, T. (2019). DistilBERT, a distilled version of BERT. NeurIPS 2019 EMC2 Workshop. https://arxiv.org/abs/1910.01108

Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F. and Liu, Q. (2020). TinyBERT: Distilling BERT for Natural Language Understanding. Findings of EMNLP 2020. https://arxiv.org/abs/1910.10351

Wang, Y., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V., Tang, Z., et al. (2023). MiniLLM: On-Policy Distillation of Language Models. ICLR 2024. https://arxiv.org/abs/2306.08543

Kim, Y. and Rush, A. M. (2016). Sequence-Level Knowledge Distillation. EMNLP 2016. https://arxiv.org/abs/1606.07947