← Distillation From First Principles

Four Controlled Distillation Experiments

What do the book's E1–E4 experiments jointly establish? Four flagships with baselines and uncertainty, a Vera-level synthesis table, ten classified cross-experiment findings, and an explicit account of what remains unmeasured.

Forty chapters built the instruments. This one reads the measurements — all of them, in one place, with the failures kept in and the denominators attached:

What do the four experiments jointly establish, and what remains unmeasured?

The short answer first, because the chapter earns it row by row: distillation reliably transfers fidelity — students learn to resemble their teachers on the agreement measures in most families tested — while capability transfers only under narrow, named conditions, and fails outright in most agentic settings tried here. That sentence is the book’s scientific conclusion in miniature. Everything below is the evidence that constrains it, the uncertainty that bounds it, and the list of what would be needed to say more.

This chapter reports established book results and labels transferred research claims as such. No new training was run, no frozen record was altered, and no experiment is reported as more successful than its manifest allows. Every figure below carries its run identity; the reconciliation test in tests/test_ch41_evidence.py re-asserts each headline number against its frozen manifest, so a future edit that strengthens a claim without strengthening the evidence fails loudly.

E1 — Classical response and logit distillation

Question. Under a matched training budget, does soft-target or logit supervision improve a small student’s held-out correctness over hard-label training, and at what calibration and cost? Run E1-20261112T233716Z-3bdf03 (protocol 1.1), rights RM-101 (VERIFIED_PERMITTED). Teacher: MLP 2→128→128→4, 17,412 parameters, supervised. Student: MLP 2→32→4, 228 parameters. Task: four-class spiral, fixed seeds, held-out test never used for tuning.

Results. The hard-label baseline (Arm A) reaches 0.669 accuracy; the mixed-distillation student (Arm B) reaches 0.642 — a deficit of 0.027 against the baseline it was meant to beat. Teacher agreement tracks accuracy almost exactly (0.670 vs 0.643), which is itself a finding: here agreement measures fidelity, not a separate capability, and both move together. The calibration column reverses the ranking: expected calibration error 0.146 for A against 0.089 for B, with lower cross-entropy for the distilled student (0.865 vs 0.873). No output-level distillation arm beats Arm A on mean accuracy; the temperature and weight variants are descriptive, not selected.

Interpretation. The directional accuracy hypothesis was not supported — reported as such in the primary record, preserved here. What transferred was calibration and likelihood quality, at a small accuracy cost and a longer wall-clock difference the manifest records without using as a capability claim. The capacity gap, the near-perfect confident teacher, and the small optimisation budget are candidate explanations explicitly labelled as hypotheses, not findings.

Cost illumination (Chapters 31, 33). The E31 inference ledger and the E1 compression demo reframe E1’s costs: the 228-parameter student is 76× smaller than its teacher with near-identical batch-1 latency on the measured host (both overhead-bound, 0.011 vs 0.018 ms) — parameter count predicted neither the latency ratio nor the memory footprint. Post-training quantisation shrinks the distilled student’s stored weights 3.5-fold (912 → 260 bytes) with 0.997 accuracy retention and bit-identical latency; 50% unstructured pruning changes nothing stored or timed. The E1 story, fully told, is: no accuracy gain from distillation, a calibration gain, and compression wins that are storage-only — each on its own ledger, none borrowed from another.

Limitations. One synthetic task, three seeds, small MLPs; host-sensitive timings; weak-student compression baseline by frozen design.

E2 — Feature, relational, and retrieval transfer

Question. Do intermediate representation or relational losses add measurable benefit over output-only distillation on a small representation model? Run E2-20261113T224338Z-edf54e (RM-103), plus the extension-09 relational study and the E2 retrieval study (E2R-20261116T095846Z-17dd71, RM-116).

Results. Output-only KL (B) edges the hard baseline (A) by 0.006 accuracy (0.884 vs 0.877) — within noise, inconclusive. Feature matching (C: 0.891) and feature cosine (D: 0.887) move representations measurably (probe accuracy tracks task accuracy; cosine/MSE diagnostics move) without establishing a capability gain beyond the noise band. The extension-09 relational arm is the largest student improvement among the studies synthesised here: F-relational at 0.934 accuracy against the 0.877 baseline (+0.057), with relational distance/angle diagnostics improving alongside.

The retrieval study distils a cross-encoder reranker into a bi-encoder: per-seed nDCG@10 deltas of +0.038, +0.021, and −0.001 (exact: +0.0378, +0.0205, −0.0013), mean +0.019 against per-arm spreads near 0.05 — suggestive, not evidentiary, and labelled as an explicit reconciliation of the earlier single-seed characterisation, which is not repeated here.

The Chapter 9 vs Chapter 29 distinction, preserved. The relational gain above belongs to the shape-classification extension with headroom to grow. The multimodal study (Chapter 29) is a different task at a saturated ceiling: relational alignment improves (RKD 0.058 → 0.046) while accuracy stands tied at 0.991 on both arms, with a noisy-accuracy probe hinting at robustness (0.788 → 0.799) without establishing it. Same mathematics, opposite evidential situations — conflating them would manufacture either a false triumph or a false null.

Interpretation. Relations can transfer where coordinates do not (Outcome A for the extension, within its three seeds); feature matching transfers geometry without proven capability gain; retrieval distillation is inconclusive at this budget. Agreement columns throughout track accuracy, as diagnostics should.

Limitations. Synthetic shape tasks; saturated multimodal ceiling; retrieval inconclusive; no production corpus tested.

E3 — Sequence, instruction, and reasoning transfer

Question. For a small language model on bounded structured tasks, how do sequence-level, reasoning-trace, verifier-filtered, and on-policy variants compare with token-level supervised fine-tuning? Runs. Transformer baseline E3T-20261115T182053Z-5f416d (RM-109), reasoning E3R-20261115T202834Z-1765b5 (RM-110), structured E3S-20261116T115159Z-1479dc (RM-117), preference E3P-20261116T092642Z-8dd412 (RM-114), LoRA E3L-…-756182 (RM-124), plus verifier, on-policy, and precursor studies.

Results. The mixed KL transformer student edges the hard-label baseline by 0.004 exact match (0.528 vs 0.525) — noise. Verification filtering is the family’s genuine win: C-filtered reaches 0.402 functional correctness against 0.367 for unfiltered teacher targets and 0.391 for oracle supervision, on 400 held-out inputs per seed with syntax and schema validity near 1.000 — the gain is arithmetic, not formatting, at matched budgets. The preference study is the book’s cleanest proxy dissociation: DPO arms reach 0.834 preference accuracy with 0.000 exact match and token accuracy of only 0.153, against an SFT baseline of 0.482 exact — the preference objective optimised the ranking, and the DPO arms, which began from random initialisation, never learned to generate.

Reasoning traces show the teacher-forcing gap in full: forced step accuracy near 0.91 with free-generated trace validity near 0.06 and answer exact match near 0.068 — next-step prediction is not free generation, and the compounding bound predicts exactly this shape. LoRA on a random frozen base fails the task at both ranks (full fine-tuning 0.400, rank-4 0.023, rank-1 0.000 with fully invalid outputs) while a smaller fully trained model reaches 0.093 — a clean negative that bounds the adapter claim to pretrained bases, not a verdict on production LoRA.

Interpretation. Filtering the teaching signal works; matching the teacher’s outputs works within noise; matching the teacher’s preferences does not create generation; matching traces teacher-forced does not transfer free generation; adapting random weights efficiently does not adapt them at all. Each row measures something different, and the chapter refuses to average them into a verdict on “E3”.

Limitations. Synthetic JSON/arithmetic tasks; three seeds; single KD weight/temperature in the LoRA study; two-point rank sweep; no pretrained-base adapter ablation; no quantised LoRA.

E4 — Agent trajectory distillation

Question. Can a small policy learn tool-use behaviour from teacher trajectories in a deterministic synthetic environment, and what compounding errors appear at inference? Runs. E4-agent E4A-20261116T120459Z-6b1354 (RM-118), E4-experts (RM-120), E4-memory (RM-119).

Results. No primary comparison in E4 shows successful held-out episodes. Chapter 26: step agreement 0.306 with episode success 0.000 on every arm — agreeing with the teacher one step in three compounds to zero successes over three-step episodes, and the recovery probe (0.361 for the recovery arm) never composes into episodes either. The 0.306³ ≈ 0.029 bound is illustrative under independence and constant-probability assumptions; the observed zero is the measured result, not proof of the approximation — preserved exactly as reconciled in D-FP-16.

Chapter 28: routing accuracy 1.000 with routed episode success 0.000; the only student successes in the chapter are 2 of 108 episodes on a single arm — correct expert selection without successful execution. Chapter 27’s memory results are restricted to a small controlled task set: the combined memory carrier succeeds where the memory-less policy fails, a bounded demonstration of external memory beating weight changes, not a general memory claim.

Interpretation. This is the book’s strongest negative territory, reported without softening: routing expertise is not executing expertise, step agreement is not episode success, and recovery components do not compose into recoveries. The auxiliary metrics are real measurements of real sub-behaviours; presenting any of them as agent success would be precisely the proxy error Chapter 34 exists to catch.

Limitations. Deterministic synthetic lookup environment; tiny policies; no real tools, no live actions; memory tasks deliberately narrow.

Synthesis table

One row per headline comparison. Budgets matched unless stated. Uncertainty is observed spread (three seeds), never inference.

Comparison Intervention Budget Primary result Uncertainty Negative / null
E1 B vs A mixed KD vs hard labels matched updates; B longer wall-clock (recorded) acc 0.642 vs 0.669 (Δ −0.027) ±0.017/±0.036 no arm beats A on accuracy
E1 calibration same same ECE 0.089 vs 0.146 ±0.019/±0.019 accuracy cost for calibration gain
E2 B vs A output KL matched acc 0.884 vs 0.877 (Δ +0.006) ±0.07–0.09 within noise: inconclusive
E2 C/D vs A feature MSE/cosine matched acc 0.891/0.887 ±0.08 geometry moves; capability unproven
E2 ext-09 F vs A relational distance matched acc 0.934 vs 0.877 (Δ +0.057) ±0.08 largest student gain; 3 seeds only
E2 retrieval B vs A teacher-distilled ranker matched, 64 train queries nDCG@10 +0.038/+0.021/−0.001 SD ~0.05 inconclusive, not a success
E2 multimodal B vs A relational alignment matched acc tied 0.991; RKD 0.058→0.046 ceiling proxy moves at ceiling
E3-trans B vs A token KL matched exact 0.528 vs 0.525 (Δ +0.004) ±0.08 noise
E3-struct C vs B verification filtering matched functional 0.402 vs 0.367 (Δ +0.035) ±0.058/±0.036 genuine, narrow
E3-reasoning B vs A trace supervision matched answer 0.068 vs 0.093; forced-step 0.91 vs validity 0.06 ±0.01 proxy/capability split
E3-pref DPO vs SFT DPO objective matched, 600 updates exact 0.000 vs 0.482; pref 0.834 ±0.02–0.04 capability collapse
E3-peft B/D vs A LoRA r4/r1, random base matched 600 updates 0.023/0.000 vs 0.400 ±0.02–0.04 adapter fails; small-full 0.093
E4-agent B/C vs A trajectory/recovery cloning matched episode success 0.000 everywhere exact zeros step 0.306 → episodes 0.000
E4-experts routed expert routing matched routing 1.000; episodes 0.000; 2/108 best exact routing without transfer
E4-memory D vs A memory carrier matched success 1.000 vs 0.000 small task set bounded demo only
E1-compression C vs B INT8 PTQ, dequantised exec 0 extra updates retention 0.997; 912→260 B; identical latency ±0.066 storage-only win
E1-compression D vs B 50% magnitude pruning 0 extra updates within seed noise ±0.04–0.06 no speedup (dense exec)

No row is ranked against another family’s row: unrelated tasks never share a scale, and the table refuses the leaderboard the reader may want.

Cross-experiment findings

Ten questions, each classified. Directly observed means measured in a frozen manifest with a matched baseline. Supported under the tested conditions means the pattern holds across the arms that test it, within stated limits. Suggestive but inconclusive means the sign is consistent but the spread covers it. Not measured and contradicted are used without euphemism.

  1. When does teacher agreement accompany real task success? Supported under tested conditions: agreement tracks correctness in E1, E2, and E3-structured (diagnostic fidelity), and diverges informatively in E3-preference (agreement without correctness) — which is why the book reports both columns everywhere rather than choosing.
  2. When does imitation improve only an intermediate metric? Directly observed: DPO preference without exact match; forced-step accuracy without trace validity; relational alignment at an accuracy ceiling; routing accuracy without episode success. The proxy gallery of Chapter 34, reaffirmed.
  3. How does teacher quality affect student learning? Suggestive but inconclusive within this book: the near-perfect confident E1 teacher coincides with a student deficit (one setting, mechanism unproven); the verified-teacher E3 arms coincide with the filtering win (confounded with filtering itself). No general law is claimed.
  4. When does verification improve the teaching signal? Directly observed in one family: E3-structured filtering (+0.035 over unfiltered teacher targets at matched budgets). Not measured elsewhere; single-family evidence, stated as such.
  5. Why can multi-step imitation fail? Directly observed mechanism with an illustrative bound: per-step agreement multiplies into episode outcomes (0.306³ ≈ 0.029), errors are likely correlated (the measured 0.000 is consistent with that, though over 36 episodes it cannot rule out independence), and recovery components do not compose. The bound illustrates; the zero measures.
  6. When is external memory preferable to changing weights? Supported under tested conditions, narrowly: the E4-memory combined carrier succeeds (1.000) where the memory-less policy fails (0.000) on a small controlled task set with deletion verified at unit scope. Nothing beyond that scope is claimed.
  7. Which compression interventions yield measured runtime benefits? Contradicted as speedups, supported as storage: INT8 quantisation shrinks stored weights 3.5-fold with identical latency (dequantised execution named); pruning changes nothing executable. No runtime benefit was measured for any compression arm in this book.
  8. How much of the apparent improvement depends on experimental controls? Directly observed where it matters: every positive in the table above sits on matched updates, splits, seeds, and teachers — the filtering gain, the relational gain, and the calibration gain survive because the budgets do. Where budgets differed (E1 wall-clock, host timings), the book records the difference and refuses the inference.
  9. What remains unknown about scaling to larger real-world models? Not measured, comprehensively: toy tasks, tiny models, three seeds, synthetic data, no production distributions. The mechanisms (compounding, proxy collapse, reservation logic) are argued to generalise as failure shapes; no magnitude claim travels.
  10. Which proposed results require new experiments rather than stronger prose? Retrieval distillation at scale, a pretrained-base LoRA ablation, poisoning/backdoor behaviour, bias probes, refusal quality, sparse-kernel speedups, and any production deployment claim — each named, none substituted with adjectives.

Mathematical synthesis

Five objectives, what each preserves, and the independent criterion that judges it. Temperature-scaled response KL (Hinton-style, T²-scaled mixed objective) preserves the teacher’s output distribution including dark knowledge; judged by held-out correctness and calibration, not by KL achieved. Feature matching (MSE/cosine, optionally projected) preserves intermediate geometry; judged by task accuracy and linear probes, not by feature similarity.

Behaviour cloning (cross- entropy on teacher trajectories) preserves the teacher’s action distribution step by step; judged by episode success, never by step agreement — the E4 verdict in one line. Preference optimisation (Bradley–Terry/DPO against a reference) preserves the teacher’s ranking over responses; judged by exact task success, never by preference accuracy — the Chapter 22 verdict in one line. Distillation regularisation (KL penalties during continual or PEFT training) preserves prior behaviour while acquiring new; judged per-task before-and-after, never by the penalty value.

The unifying principle, stated once and used everywhere: an optimised transfer objective is evidence that optimisation ran, not that capability transferred. Every objective above can reach zero loss while the capability column reads zero — DPO and the E4 agents are the measured instances — so the book’s evaluation rule is absolute: the loss curve is process telemetry, and only

Optimising the proxy is not transferring the capability

Figure 1. Three within-family dissociations: DPO preference 0.834 against exact match 0.000; routing 1.000 against episode success 0.000; forced-step near 0.91 against free validity near 0.06. Pairs share no common scale and are never ranked — the gap within each pair is the point. Frozen E1-E4 measurements. held-out task performance under matched budgets promotes.

Scientific conclusion

What worked. Verification filtering of teaching data (E3-structured, +0.035). Relational supervision with headroom (E2 extension, +0.057). Calibration and likelihood gains from mixed distillation (E1, ECE cut by about two-fifths, 0.146 to 0.089). Bounded external memory over memory-less policies (E4-memory, small scope). Storage compression with full retention (INT8, 3.5-fold).

What failed. Accuracy gains from classical KD (E1 deficit preserved as a negative). DPO as a capability method (zero exact match measured; generation never formed). Trace supervision for free generation (gap measured). LoRA on random bases (floor measured). Agent episode success across all primary comparisons (zeros measured). Routing as a substitute for execution (1.000 vs 0.000 measured). Speedups from quantisation or pruning on the executed kernels (nulls measured).

What was inconclusive. Output-KL and feature-matching capability deltas; retrieval distillation at this budget; multimodal robustness hints; transformer-KL edges; diffusion distillation advantage over naïve step-skipping.

What was computationally efficient. Nothing that runs. INT8 storage (3.5-fold smaller checkpoints), the E31 cost ledger (same-host measurement replacing parameter-count guessing), and LoRA’s optimiser-state saving (5.8-fold) are budget facts, not deployment speedups — the book keeps that boundary on every page that mentions them.

What additional evidence would be required. Larger tasks with headroom, pretrained-base ablations, production-scale retrieval corpora, poisoning and bias studies, refusal measurement, sparse-kernel benchmarks, and any deployment claim whatsoever — each a designed study with its own rights record, not a paragraph.

No fourth successful experiment was fabricated for narrative symmetry. The E4 column reads zero because the manifests read zero, and the chapter ends the science there.

Reproducibility check from a manifest

The contract requires one, performed read-only. Take E3S-20261116T115159Z-1479dc: run identity, protocol 1.0, config hash, eight source hashes, RM-117 VERIFIED_PERMITTED for train_structured_distillation_e3, three seeds with per-seed source fingerprints, per-arm functional correctness with agreement reported separately, limitations recorded. A reviewer can verify every cell of the E3-structured row above from this manifest alone — and the Chapter 41 reconciliation test does exactly that check in code, pinning 0.3908/0.3667/0.4017 against the frozen file.

What the reviewer cannot do is re-derive the numbers without re-running training, which the programme prohibits to protect frozen evidence; the check therefore attests presence and consistency, never recomputation. That boundary — reproducible records, not reproduced runs — is stated on the record rather than smuggled past it, and it applies identically to all four flagships.

Rights completeness per flagship

Every experiment references its rights record, and the restrictions travel with the results. E1 trains and evaluates under RM-101; E2 representation under RM-103; E3 studies under their per-study records (RM-109 transformer, RM-110 reasoning, RM-114 preference, RM-117 structured, RM-124 LoRA); E4 under RM-118/119/120; retrieval under RM-116 — every status VERIFIED_PERMITTED for its recorded training-adjacent action, reviewed by a named author. Redistribution, commercial use, and publication are authorised nowhere in these records: the Chapter 36 twelve-action audit stands, and this chapter publishes no outputs beyond aggregate metrics already versioned in the frozen manifests. No checkpoint, trajectory, or training row leaves the repository through this chapter.

The cost lens (Chapters 31–33, applied)

The synthesis is incomplete without the efficiency chapters, because capability without cost is a brochure. Chapter 31’s ledger discipline — measure latency, memory, and storage on one host rather than inferring them from parameter counts — is what lets the table report training and inference costs as measurements instead of adjectives. Chapter 32’s ledger separates trainable from deployed: the LoRA student’s 768 trainable parameters ship inside 5,232 deployed ones, which is why the capability row and the efficiency row can disagree without contradiction.

Chapter 33’s ledger separates stored from executed: 3.5-fold smaller checkpoints with identical latency is a storage result wearing no speed costume. The joint finding across all three: nothing in this book got faster, one thing got smaller, one thing got cheaper to optimise — and the capability conclusions above stand unchanged whichever ledger the reader cares about, because none of them was ever borrowed from another.

Limitations

Three seeds throughout; descriptive statistics only, no significance claimed. Synthetic tasks; tiny models; no transfer to production distributions asserted. Frozen evidence re-read, never re-measured; hashes recorded, never re-verified. The reconciliation test pins the cited figures; anything not pinned is not claimed. No independent review; this chapter reports evidence, not acceptance.

Preparing for deployment governance

Evidence complete, verdicts mixed, one column of zeros. The remaining question is governance, not science: given students like these — some capable, some broken, all unapproved — how does anything ever ship safely, or rightly stay unshipped? Chapter 42 builds the promotion gate, the monitoring, and the rollback that answer it.

References

Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H. and Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM 64(12). arXiv:1803.09010. https://arxiv.org/abs/1803.09010

Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D. and Gebru, T. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT 2019).* https://doi.org/10.1145/3287560.3287596