Evaluating Knowledge Transfer
What evidence justifies the claim that a student acquired a capability from a teacher? A transfer taxonomy, the mathematics of uncertainty, and a synthesis of every E1–E4 finding through one evaluation lens.
Every experiment in this book reports two numbers that look alike and mean different things: how often the student is right, and how often the student agrees with the teacher. Thirty-three chapters have kept them in separate columns. This chapter explains why that separation is the whole of evaluation — and what else must accompany it before anyone may claim that a capability transferred:
What evidence justifies the claim that a student acquired a capability from a teacher, rather than merely fitting a dataset or copying a teacher?
The honest answer is a conjunction, never a single score. The student must beat an appropriate baseline on held-out task data, with the budget matched and the uncertainty stated; agreement must be reported separately as a diagnostic; calibration, slices, contamination, and cost must be checked; and the classification of the outcome must be argued in prose, not read off a threshold. The chapter builds that conjunction from first principles, implements it as a small read-only assessment layer, and then turns it on every major finding in the book — including the ones that fail it.
The taxonomy: five outcomes, no autopilot
Chapter 05 defined independent correctness (KD-032) against teacher–student agreement (KD-031). The evaluation taxonomy extends that distinction into five mutually exclusive outcomes:
Outcome A — Successful capability transfer. The student achieves independently verified task success under the declared evaluation conditions: held-out data, matched baseline, stated uncertainty. The teacher’s involvement explains how the student was trained; the held-out score explains what the student can do.
Outcome B — Partial transfer. The student learns a measurable component of the behaviour without the complete capability: template without arithmetic, recovery without episodes, relational structure without accuracy.
Outcome C — Proxy transfer. An intermediate metric moves — agreement, preference accuracy, embedding similarity, verification scores — without measured independent capability improvement. The proxy is real; the capability is not demonstrated.
Outcome D — Transfer failure. The student fails to acquire the capability or regresses materially against the baseline. A clean negative, reported, not hidden.
Outcome E — Inconclusive. The evidence cannot distinguish transfer from noise, confounding, or insufficient measurement: one seed, mismatched budgets, missing denominators, contaminated splits.
The categories are argued, not computed. The assessment layer in
examples/common/evaluate.py proposes an outcome from the deltas
but refuses to classify without a written justification of at least
a sentence: classification without reasoning is the failure mode the
module exists to prevent. A threshold never promotes a result; a
reader does, on the record.
What each metric establishes
Accuracy is the fraction of held-out predictions matching ground truth — the primary capability metric wherever labels exist. Exact-match and episode-level success are its stricter siblings for sequences and agents: the whole output must be right, the whole episode must succeed. Teacher agreement is the fraction of predictions on which student and teacher agree — a diagnostic of fidelity, never a measure of correctness.
Stanton et al. (P-062) show why the separation matters beyond this book: student–teacher fidelity gaps persist even when the student has capacity to match, and closer fidelity does not always improve generalisation. Their result is peer-reviewed and architecture-specific, cited here for the principle, not the numbers: agreement and correctness can move independently, so any evaluation that reports one as the other is broken by construction.
Calibration (KD-030) asks whether confidence means what it says. Expected calibration error bins predictions by confidence and averages the absolute gap between bin confidence and bin accuracy; the Brier score is the mean squared error against the one-hot label and decomposes, roughly, into calibration plus refinement. A worked miniature fixes the idea: ten predictions with confidences 0.9 × 8 and 0.34 × 2, of which only four are correct, carry ECE above 0.6 — confidence high, correctness low, the gap measured rather than eyeballed.
Fixture C in Chapter 35 computes exactly this record (accuracy 0.400, ECE 0.652). Label smoothing (Müller et al., P-042) is the cautionary neighbour: smoothing improves calibration while making the teacher’s logits less informative for distillation — the same intervention that repairs one column degrades another, which is why the book reports both. Retrieval adds rank-aware metrics — nDCG, MRR, recall at k — each with a stated k and a stated corpus size, because a recall without a denominator is a rumour.
Uncertainty is reported before any comparison is drawn. With three seeds the book reports mean and standard deviation as description, never as inference: no significance is claimed, and the text says so. Where an interval helps, the assessment layer computes a seeded percentile bootstrap of the mean — resample the observed values with replacement, take the mean each time, read off the percentiles — with the interpretation fixed in prose: the interval quantifies the observed spread; with n = 3 it confers no generalisation. A worked instance: the E1 arm-B accuracies 0.642 ± 0.017 resample to a 95% interval roughly spanning the observed values themselves, because with three points the resampled means cannot reach beyond them — the bootstrap honestly reports that there is almost no information about the tails, which is precisely why the chapter refuses to upgrade description to inference.
A single seed is a limitation, not an interval: the module raises rather than invent a spread. For low-count successes the chapter states the denominator rule plainly. Eighteen of thirty-six test episodes succeeding is a rate of 0.500 on n = 36. Thirty-six of thirty-six is not proof of perfection: with zero observed failures the failure rate is bounded below roughly 3/n at about 95% confidence (the rule of three), not at zero. Never use zero observed failures as proof of a zero failure probability.
Multi-step behaviour compounds. If each step is correct with
probability p and errors propagate, whole-trace validity is bounded
above by roughly p raised to the number of dependent steps — the
reason Chapter 18’s forced step accuracy near 0.91 coexists with
free-generated trace validity near 0.06 without contradiction. The
bound is an inequality, not a prediction: recovery, verification,
and correlated errors all move the realised value, which is why the
book measures traces rather than multiplying step rates. Cost joins
the comparison through Pareto dominance from Chapter 31: a student
is comparable to a baseline only on shared tasks with shared
denominators, and the assessment layer refuses cross-task
aggregation outright, returning NOT_COMPARABLE instead of a
ranking.
The assessment layer
examples/common/evaluate.py is deliberately read-only. It defines
EvaluationRecord — experiment, run id, arm, metric, mean, spread,
seed count, denominator, task, source, limitations — and a set of
refusals. Unknown metrics raise. Out-of-range values raise.
Non-finite values raise. Missing denominators raise. Cross-task or
cross-population comparisons return NOT_COMPARABLE. Absent
measurements are NOT_MEASURED; insufficient ones INCOMPLETE.
Contamination is checked by identifier overlap: any shared id
between train and test voids the split, and the check reports the
overlap rate rather than a pass.
Slice regressions are flagged by
comparing overall and slice deltas: an average that improves while
a critical slice regresses returns masked_regression = True, and
the worked table below shows what that looks like. Nine focused
tests pin the math on known-answer fixtures, including a spot
reconciliation against frozen manifests — E1 arm A at 0.669
accuracy, E4 trajectory success at exactly zero, the DPO arms at
0.834 preference with 0.000 exact match — so that any future drift
in the cited records fails loudly instead of silently.
A worked evaluation report: E1–E4 in one ledger
The contract requires an evaluation report for E1–E4. Here it is — one row per headline comparison, each with its denominator, budget note, and outcome under the taxonomy. Every figure below is read from the frozen manifests cited; the assessment layer’s spot test pins three of them.
| Comparison | Capability delta | Agreement | Denominator | Budget | Outcome |
|---|---|---|---|---|---|
| E1 B-mixed vs A-hard | −0.027 accuracy | 0.643 vs 0.670 | 1,000 test, 3 seeds | matched updates; B wall time longer | D on accuracy; A on calibration (0.089 vs 0.146) |
| E2 B-output-KL vs A-hard | +0.006 accuracy | +0.008 | shape test, 3 seeds | matched | E (within noise) |
| E2-repr F-relational (Ch09 ext.) vs A-hard | +0.056 accuracy | +0.057 | shape test, 3 seeds | matched | A (largest student gain in the book) |
| E3-transformer B-KL vs A-hard | +0.004 exact match | tied | instruction test, 3 seeds | matched | E (within noise) |
| E3-structured C-filtered vs B-teacher | +0.035 functional | +0.032 | 400/seed, 3 seeds | matched | A |
| E3-reasoning B-trace vs A-answer | −0.025 answer exact | n/a (different outputs) | chain test, 3 seeds | matched | C for traces; D for full validity |
| E3-preference B-DPO vs A-SFT | −0.482 exact match | preference 0.834 vs 0.984 | instruction test, 3 seeds | matched | D for capability; C for preference |
| E4 B-trajectory vs A-answer | 0.000 vs 0.000 success | step 0.306 | episodes, 3 seeds | matched | D |
| E4-experts C-routed vs A-single | 0.000 vs 0.056 success | routing 1.000 | episodes, 3 seeds | matched | D; routing column is C |
| E2-multimodal B-rel vs A-signal | 0.000 (ceiling) | tied | shape+signal test | matched | C (proxy moves, ceiling binds) |
| D30 S8 vs N8-naive | 0.000 quality | n/a | mixture test | matched | E for distillation; A for step cut |
| E2-retrieval B-teacher vs A-contrastive | +0.019 nDCG@10 | +0.021 at rank 1 | corpus queries, 3 seeds | matched | E (within noise) |
Three patterns deserve emphasis because they recur. First, the budget column is uniformly “matched”: every comparison above holds updates, splits, and teacher fixed while varying one factor — the causal attribution the evidence standard demands. Where a delta exceeds its seed spread, it is attributable to the named intervention rather than to confounded effort; where it does not, the row is Outcome E. Where budgets did differ (E1’s longer arm-B wall time, the E31 host-sensitive timings), the book records the difference and refuses the capability inference.
Second, the agreement column never decides: in no row does agreement rescue or overturn the capability verdict, which is the schema working as designed. Third, the E rows outnumber the A rows — and that is the point of the exercise. A report in which every intervention works is a report that cannot detect failure; this one records failure or inconclusiveness in ten of twelve headline comparisons. The two classed as Outcome A are the relational extension (Chapter 09) and filtering, whose delta over unfiltered targets is positive on every seed.
Synthesis I: the proxy gallery
The book’s negative results are its most instructive, because each isolates one way a proxy can move while capability stands still.
Chapter 18 (E3-reasoning). Forced step accuracy 0.908 against free-generated trace validity 0.063 and answer exact match 0.068. Teacher-forcing measures next-step prediction with the truth supplied; free generation measures the behaviour. The gap is not a mystery to be explained away — it is the measurement working: teacher-forced accuracy is a proxy (Outcome C for the trace claim), free validity is the capability column (Outcome D for full traces, B for steps). The chapter’s compounding bound predicts exactly this shape.
Chapter 22 (E3-preference). DPO arms reach 0.834 preference accuracy with 0.000 exact match and token accuracy collapsing to 0.153. Preference agreement moved; generation never arrived, because the DPO arms started from random initialisation rather than from the SFT model. This is Outcome C in its purest form — the preference objective optimised the ranking without ever producing a working generator — and the book’s refusal to report preference accuracy as capability is the evaluation discipline paying off. The SFT baseline at 0.482 exact match stands as the capability reference the DPO arms never touch.
Chapter 26 (E4-agent). Step agreement 0.306 with episode success 0.000 on every arm. Agreeing with the teacher one step in three, compounded over three-step episodes with no recovery, is consistent with zero successes — the compounding bound again, measured. Outcome D for episodes; Outcome B at most for isolated steps, and only where the denominator supports it.
Chapter 28 (E4-experts). Routing accuracy 1.000 with routed episode success 0.000. Correct expert selection is a proxy for the capability the system was built to deliver; the student executes neither specialty. Outcome C for routing-as-capability, D for the deployed claim. The contrast with Chapter 26 is deliberate: agreement can fail at the step level or succeed at the routing level, and both leave episode success at zero — two different proxy failures, one capability verdict.
Chapter 29 (E2-multimodal). Relational alignment improves (RKD 0.058 → 0.046) while accuracy stands at 0.991 on both arms — a saturated task where the representation moved and the capability had nowhere to go. Outcome C with an asterisk: the proxy gain is real and measured, the capability column is at ceiling, and the chapter refuses to manufacture a distinction the denominator cannot support. The noisy-accuracy probe (0.788 → 0.799) hints at robustness without establishing it — Outcome E, stated as such.
Chapter 30 (D30-diffusion). Quality metrics tied across the teacher and all students (MMD 0.101–0.102, coverage 1.000 everywhere) while latency halves with each halving of the step count — but the naïve step-skip matches the distilled sampler, so no distillation advantage is demonstrated. Outcome E for the distillation claim (the null is preserved, not rejected), Outcome A for step reduction as pure engineering. The negative control — naïve skipping — is what makes the verdict possible; without it the latency win would have been misattributed to distillation.
Chapter 24 (E2-retrieval). The teacher-distilled retriever leads the contrastive baseline by +0.019 mean nDCG@10 — with per-seed deltas of +0.038, +0.021, and −0.001 against per-arm standard deviations near 0.05. Two seeds positive, one flat, all within noise: Outcome E, stated plainly. The agreement column (0.271 vs 0.250 at rank 1) moves with the ranking, as a diagnostic should; it does not rescue the capability claim. This is the chapter’s correction to a looser reading: a consistent sign across two of three seeds is suggestive, not evidentiary, and the book declares it inconclusive rather than rounding it up.
Synthesis II: the capability ledger
Positive and partial results get the same treatment — deltas with denominators, budgets matched, limitations attached.
Chapter 25 (E3-structured). Verification filtering (C) reaches 0.402 functional correctness against 0.367 for unfiltered teacher targets (B) and 0.391 for oracle supervision (A), on 400 held-out inputs per seed across three seeds. Syntax and schema validity sit near 1.000 throughout, so the gain is arithmetic, not formatting. C beats B on every seed (+0.010/+0.038/+0.058); its lead over A reverses on one seed, so C > A is not established. Outcome A for filtering against unfiltered targets, with the three-seed spread (±0.058 on C) stated. The budget is matched by construction: same splits, same updates, same teacher.
Chapter 31 (E31-benchmark). Selection under explicit constraints — accuracy ≥ 0.64 with ECE ≤ 0.10 — picks the mixed-distillation arm (0.642/0.089) over the accurate but miscalibrated baseline (0.669/0.146). The evaluation contribution is the procedure: thresholds composed, qualifying sets reported rather than winners, Pareto frontiers computed per family with costs tied by construction. Outcome A for the selection method; the arm ranking is an instantiation, not a leaderboard.
Chapters 32–33 (E3-peft, E1-compression). Full fine-tuning 0.400 against LoRA r=4 at 0.023; quantisation retaining 0.997 of a weak baseline with latency identical to the nearest 0.001 ms. Both are Outcome A for their measurement claims (the ledger entries are exact) and Outcome D for the capability hopes they extinguish (adapters that fail, speedups that never appear). A negative result with a matched budget and a stated denominator is evidence, not absence — the evidence standard’s reporting rule, applied.
Calibration, slices, and contamination
The calibration table the contract requires is frozen E1, and it
carries the book’s recurring trade-off in four rows: arm A, 0.669
accuracy at 0.146 ECE; arm B, 0.642 at 0.089; arm C, 0.629 at
0.068; arm D, 0.579 at 0.112. Calibration does not track accuracy
here: the most accurate arm, A, is the worst calibrated — the reason
the book reports both and lets the thresholds decide. The subgroup
table is the masked-regression pattern with the flag computed in
the assessment layer: overall accuracy +0.060 while the critical
slice falls −0.200, masked_regression = True, promotion blocked.
The records are synthetic and labelled as such; the mechanism is
real, and Fixture A in Chapter 35 reuses it as a regression test.
The contamination audit reports a clean result with the procedure
shown: every training study in E1–E4 asserts disjoint splits at
runtime (assert_disjoint), corpora carry fingerprints (the E2
retrieval corpus fingerprint is pinned in its manifest), and the
assessment layer’s overlap check stands ready with a synthetic
positive control (one shared id in five test items: status
CONTAMINATED, overlap 0.200). No contamination was found; the
chapter claims the audit, not the impossibility.
Reporting follows the model-card practice (Mitchell et al., P-044): intended use, disaggregated performance, evaluation data, and caveats — with the dataset-datasheet habit (Gebru et al., P-043) behind the provenance columns. A complete results report, on this pattern, states the task and population, the baseline and its budget, the primary capability metric with its denominator, the agreement diagnostic, calibration where confidence is exposed, uncertainty as observed spread, slices with any masked regression, the contamination audit, the cost ledger, and every limitation — including the failed seeds, of which this book preserves many (D at 0.000 in E3-peft, C-combined at 0.000 in E4-experts, the DPO arms at 0.000 exact match).
Evaluation data has rights of its own: benchmark licences may restrict redistribution, and personal data in evaluation sets invokes GDPR obligations (L-021) — the reason the book’s evaluation sets are synthetic and the chapter records that choice as a rights decision, not a convenience. Human benchmarks reused from elsewhere carry contamination risk by default (the HumanEval caution, Chen et al., P-057): widely evaluated-on data may live in some teacher’s training set, and a held-out split of a contaminated benchmark is held out in name only. The book sidesteps this with self-created tasks; anyone who does not must audit before claiming.
Failure modes of evaluation itself
Six anti-patterns, each with the book’s own near-miss or catch. Agreement presented as accuracy: refused by schema (separate columns, separate metrics). Test-set tuning: refused by protocol (best-validation checkpoints, frozen test use — Chapter 15’s discipline). Single-seed overclaim: refused by the bootstrap guard (one value raises; the text calls it a limitation). Averaging away a subgroup regression: flagged by the slice check, with the E4 routing case as the standing example.
Missing baseline budget: stated wherever budgets differ (E1’s longer arm-B wall time is recorded, not used). Ignoring contamination: audited above. The common root is presenting a measurement as a conclusion; the chapter’s remedy is to keep the two visibly apart — numbers in tables with denominators, conclusions in prose with justifications, and the assessment record linking them.
Limitations
Three seeds throughout: every comparative claim is descriptive, and the chapter never upgrades description to inference. Task coverage is synthetic and narrow; nothing here transfers to production distributions without re-measurement. The bootstrap intervals describe spread, not truth. Contamination auditing covers identifier overlap, not semantic leakage (paraphrased test items would pass the check and void the spirit). Calibration is reported, not repaired: no recalibration method is evaluated. No external replication and no independent review.
Preparing for failure modes
Evaluation tells the reader what the student learned. It does not tell the reader what the student breaks — the rare slice that regresses, the confident error, the invalid tool call, the teacher bias faithfully preserved, the permission that was never granted. Chapter 35 turns the assessment layer’s refusals into a regression suite: the same slice flag, run before every promotion, with teeth.
References
Müller, R., Kornblith, S. and Hinton, G. (2019). When Does Label Smoothing Help? Advances in Neural Information Processing Systems (NeurIPS 2019). arXiv:1906.02629. https://arxiv.org/abs/1906.02629
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H. and Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM 64(12). arXiv:1803.09010. https://arxiv.org/abs/1803.09010
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D. and Gebru, T. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT 2019).* https://doi.org/10.1145/3287560.3287596
Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A. A. and Wilson, A. G. (2021). Does Knowledge Distillation Really Work? Advances in Neural Information Processing Systems (NeurIPS 2021). https://papers.nips.cc/paper_files/paper/2021/hash/376c6b9ff3bedbbea56751a84fffc10-Abstract.html
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J. et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374. https://arxiv.org/abs/2107.03374
Legal sources
Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data (General Data Protection Regulation). Applicable 25 May 2018. https://eur-lex.europa.eu/eli/reg/2016/679/oj (retrieved 12 November 2026). Cited as L-021 for the evaluation-data rights and minimisation basis; not legal advice.