← Distillation From First Principles

On-Policy Distillation

Train the student on its own generated trajectories with teacher supervision: forward and reverse KL, the distribution-mismatch argument, detached-prefix gradient surrogates, and a controlled E3 comparison of offline, on-policy, and stale-replay distillation.

Chapters 17 through 19 trained the student on fixed collections: oracle answers, oracle traces, teacher-generated candidates, verifier-selected subsets. The training states never changed, no matter how the student changed. This chapter asks what happens when they do:

Does supervising the student on its own generated outputs improve the match between training and inference distributions, and at what cost?

The honest answer, on our task, is two-sided. Matching the prefix distribution does nothing measurable for correctness — the on-policy student ties the offline student to three decimals (0.452 versus 0.453). But the choice is not irrelevant: training on samples drawn once from the untrained initial student and replayed throughout collapses correctness from 0.45 to 0.08. Those samples were both stale and near-garbage, so the experiment cannot say which of the two did the damage. Where the states come from matters enormously when they are wrong and stale, and not measurably, here, when they are fresh. That asymmetry is the chapter’s finding, and the machinery below exists to make it checkable rather than merely plausible.

Why offline distillation misses student errors

A student trained by teacher forcing learns conditional distributions: given a reference prefix, predict the next token. At inference it conditions on its own prefixes, which may wander into states the reference data never visits. An early mistake compounds — each subsequent prediction conditions on the mistake — which is the mechanism behind exposure bias (Bengio et al., 2015; Arora et al., 2022, P-016 register). Offline distillation supervises exactly the states of a fixed dataset; it never sees the student’s own error states, so it cannot correct them. On-policy distillation (P-015, P-016) samples from the student, scores those states with the teacher, and trains there. Whether that helps is empirical; that it changes where training happens is definitional.

KD-024 — On-policy distillation

On-policy distillation (KD-024): student-generated trajectories are supervised by a teacher, reducing the mismatch between training and inference distributions. The defining property is the source and freshness of the sampled states: prefixes drawn from the current student policy, scored by the teacher, used within the same training window. A dataset generated once by an older checkpoint and reused for hundreds of updates is not on-policy relative to a changing student — it is replay, and Example D below shows why the distinction matters.

KD-025 — Off-policy distillation

Off-policy distillation (KD-025): a fixed teacher-generated dataset is used independently of the student’s current policy. Teacher traces (Chapter 18), verifier-selected candidates (Chapter 19), and ground-truth answers (Chapter 17) are all off-policy in this sense: the behaviour policy that produced the states — oracle, teacher, or verifier filter — is not the student being trained. Record the actual behaviour policy wherever known; “offline” alone does not say whose outputs the student sees.

Which model generates the training states?

Two pipelines, different state sources. Off-policy: a fixed behaviour policy generates data; data is stored; the student trains against the fixed distribution. On-policy: the current student generates prefixes; the teacher scores those states; the student updates; the sampling distribution evolves with the student. The reader must keep one fact in view: the key difference is where training states come from, not whether a teacher is present. Both pipelines in this chapter use the same frozen teacher and the same conditional loss. Only the prefix source differs — which is exactly what makes the comparison a test of KD-024 rather than a test of everything at once.

    flowchart LR
  F[fixed behaviour policy] --> D[stored dataset]
  D --> S1[student trains on fixed states]
  S2[student samples own prefixes] --> T[teacher scores those states]
  T --> U[student updates; distribution evolves]
  

Figure 1. Off-policy (top) reuses one fixed distribution; on-policy (bottom) trains where the student currently stands. Same teacher, same conditional loss — only the state source differs.

Forward KL

Let p_T(y|x) be the teacher distribution and p_S(y|x) the student distribution over outputs y given input x. The forward KL is:

\(D_{KL}(p_T \mid\mid p_S) = \sum_y p_T(y|x) \log \frac{p_T(y|x)}{p_S(y|x)}\)

The expectation is teacher-weighted: every output counts in proportion to its teacher probability. Where the teacher assigns mass, the student must also assign mass or pay steeply — forward KL is zero-avoiding on the student side, which is why it is mode-covering: a student that cannot represent all teacher modes stretches across them, assigning unreasonable probability to regions the teacher rates near zero (Huszár, 2015; MiniLLM §2, P-015). It corresponds, under an empirical data distribution, to maximum likelihood — the familiar cross-entropy objective is forward KL plus a constant teacher entropy. Support condition: wherever p_T > 0 the student must keep p_S > 0, or the divergence is infinite. Missing a teacher mode is the expensive error.

Reverse KL

\(D_{KL}(p_S \mid\mid p_T) = \sum_y p_S(y|x) \log \frac{p_S(y|x)}{p_T(y|x)}\)

Now the expectation is student-weighted: outputs count in proportion to student probability. Probability the student places where the teacher assigns little is penalised directly — reverse KL is mode-seeking: the student concentrates on teacher modes it can represent and ignores the rest. Support condition: wherever the student puts mass, the teacher must have mass, or the penalty is unbounded. The failure it invites is the mirror image: collapse onto one mode while the task needs several. Do not claim reverse KL always causes mode collapse; it prefers concentration, and whether that hurts depends on whether the ignored modes matter for correctness.

Mode covering and mode seeking

Example A makes the asymmetry concrete. Teacher T = [0.7, 0.2, 0.1], student S = uniform [1/3, 1/3, 1/3]. Forward KL(T||S) = 0.7·ln(2.1) + 0.2·ln(0.6) + 0.1·ln(0.3) ≈ 0.297. Reverse KL(S||T) = (1/3)·ln(0.476)

  • (1/3)·ln(1.667) + (1/3)·ln(3.333) ≈ 0.324. Different numbers, different incentives: the forward direction charges 0.7·ln(2.1) for under-covering the teacher’s main mode; the reverse direction charges (1/3)·ln(3.333) for the student’s mass on the teacher’s weakest outcome.

On open-ended generation with limited student capacity, the forward penalty forces coverage of long-tail teacher variants the student will then emit at inference (the MiniLLM §1 argument, P-015); the reverse penalty forbids exactly those emissions at the price of neglected modes. Neither direction is universally correct; each buys one failure mode and sells the other.

Direction is a choice of failure mode, not a better loss

Figure 2. Example A drawn as bars: the same teacher-student pair scores 0.297 forward and 0.324 reverse, charging different mistakes. Worked arithmetic, test-pinned, not measured data.

Autoregressive sequence probabilities

For an output sequence y = (y_1 … y_L):

\(p(y|x) = \prod_{t=1}^{L} p(y_t \mid x, y_{

Each factor is a token-level conditional on the prompt plus the preceding prefix. Teacher forcing trains these conditionals on reference prefixes; free generation evaluates their composition on self-produced prefixes. Masks and denominators must be explicit: loss averages over supervised positions only (completion span, never PAD), and the sequence log-probability is the sum of conditional log probabilities — a fact the tests verify by both routes (test_autoregressive_factorisation). Variable length is handled by EOS: positions after EOS contribute nothing, and rows carry PAD to a fixed length that masking excludes.

Teacher prefixes versus student prefixes

Sequence-level forward KL weights prefixes by the teacher distribution; sequence-level reverse KL weights them by the student distribution. That weighting is the connection to state visitation: forward-KL training supervises teacher-visited states, reverse-KL training supervises student-visited states. The decomposition is exact. Write sequence forward KL by factoring the first token:

\(D_{KL}(p_T(y|x) \mid\mid p_S(y|x)) = E_{y_1 \sim p_T}[\log p_T(y_1|x)/p_S(y_1|x)] + E_{y_1 \sim p_T}[D_{KL}(p_T(\cdot|x,y_1) \mid\mid p_S(\cdot|x,y_1))]\)

and recurse: at every position, the conditional divergence is averaged over prefixes drawn from the teacher’s prefix distribution. Swapping the roles of p_T and p_S throughout — student prefixes outside and the reverse conditional KL(p_S‖p_T) inside — yields sequence reverse KL. Swapping only the outer sampler, while keeping the forward conditional, does not. Two consequences follow. First, teacher-prefix and student-prefix supervision can behave differently even with identical conditional losses — the same per-token divergence, averaged over different prefix distributions, is a different objective.

Second, our surrogate (conditional forward KL at detached student prefixes) shares only the outer weighting of the reverse decomposition — student prefixes — while keeping the forward conditional inside and dropping the gradient through sampling — precisely the gap the six-point documentation rule below labels. Example B: a teacher trace for 7 + 4 then × 3 reads STEP 7+4=11, STEP 11×3=33. A student that instead generates STEP 7+4=12 will, under offline training, never be supervised at the state “12 × 3” — yet at inference that is exactly where it stands. On-policy training puts the teacher’s conditional (what follows “12 × 3”?) at the student’s actually-visited prefix. Whether the student benefits is the experiment, not the derivation.

Exact objectives and practical gradient surrogates

This is the chapter’s critical technical point. Suppose the student generates prefixes and we compute token-level teacher/student KL at those prefixes, treating the sampled prefixes as detached observations. The gradient then does not account for how the prefix distribution changes with student parameters — the sampling step is stopped-gradient. That implementation is not the exact gradient of sequence-level reverse KL. In particular: student-generated prefixes combined with conditional forward KL are not exact sequence-level reverse-KL optimisation. Exact reverse-KL gradients need the score-function term through the sampling distribution (MiniLLM Eq. 2, P-015); GKD-style detached conditionals (P-016, Eq. 4) deliberately omit it for stability.

Document separately: (1) which policy produces prefixes, (2) which conditional divergence direction, (3) which parameters receive gradients, (4) whether sampling gradients are included, (5) the actual estimator, (6) the approximated objective. An accurately labelled stop-gradient conditional-KL surrogate is acceptable for the bounded experiment. Our primary objective is therefore stated plainly: an on-policy prefix-distribution surrogate using conditional forward KL — student-sampled prefixes, forward conditional direction, gradients through student logits only.

Building the on-policy loop

examples/instruction_distillation/on_policy.py implements the minimal loop on the shared workbench: sample continuations from the current student (temperature multinomial, seeded per step, EOS and length enforced), assemble prompt+sample rows with a sampled-span mask, score with the frozen teacher (TeacherScorer, query/token ledger), compute conditional forward KL on the span, backpropagate through the student only, clip, step, and checkpoint by hard CE on the oracle validation split after the full budget. Replay buffer stores rows with their sampling version; only version == current step counts as fresh. Failures are explicit: empty samples, over-length contexts, and non-finite losses raise instead of silently continuing.

Sampling from the current student

Example C is the actual training step (abridged; the runner adds accounting):

gen = torch.Generator().manual_seed(seed * 1_000_000 + step)
idx = torch.randint(0, n_train, (32,), generator=gen).tolist()
prows = [prompts[i] for i in idx]
conts, ledger = on_policy.sample_continuations(
    student, torch.tensor(prows), max_new=32, temperature=1.0,
    generator=gen, eos_id=rsn.EOS_ID, max_length=64)
rows, mask = on_policy.rows_from_samples(prows, conts, pad_id=0, row_length=64)
opt.zero_grad(set_to_none=True)
s_logits = student(rows[:, :-1])
t_logits = scorer.score(rows[:, :-1], mask=mask[:, 1:])
loss = losses.token_kl(s_logits, t_logits, rows[:, 1:], temperature=1.0, mask=mask[:, 1:])
on_policy.check_finite(loss, what="student loss")
loss.backward()

Sampling must come from the current parameters — the sampler runs in eval/no-grad mode on the live student object, and the provenance record carries policy version, step, seed, and temperature. Reusing an old checkpoint’s outputs here would be replay mislabelled as on-policy; the buffer’s version field exists to make that fraud structurally visible.

Teacher supervision on student states

For each sampled prefix the frozen teacher supplies full conditional logits — not just an answer label. The teacher runs under no-grad, is switched to eval once, and a test asserts its parameters are bit-identical before and after scoring while student gradients are nonzero and no teacher gradient exists. Masking covers sampled positions only: prompts are given, not learned here. If a sample cannot be scored (empty continuation, over-length context), the run raises an explicit error rather than fabricating probabilities. No future-token leakage: teacher and student see identical shifted contexts, and the mask aligns prediction positions exactly.

Stale trajectories and replay

Example D: at step 0 the student is randomly initialised, so its samples are near-garbage token strings; at step 500 the same prompts elicit structured traces. A buffer frozen at version 0 and reused at step 500 supervises states the student no longer visits, with a staleness age of 500. Arm C is exactly this: one sample per training prompt from the initial student, resampled in batches, age growing 0 to 600. GKD’s λ parameter (P-016) spans this spectrum deliberately — λ=1 pure on-policy, λ=0 supervised, mixtures between. Our C is the λ=0-in-spirit extreme with student-generated (not teacher) fixed data, which confounds staleness with sample quality, because the frozen samples come from the untrained student.

The frozen E3 experiment

Run E3O-20261116T073939Z-3dbca7, protocol 1.0, rights RM-112 (generate_student_trajectories_and_train_on_policy_e3, verified-permitted, gate first). Fresh deterministic splits, seeds 10, 11, 12: train 800, val 200, test 600, ood 400; 2-step train, 3-step OOD. Teacher w32/L2, 800 updates, oracle trace+answer mask (greedy fully-valid 87/100 on train prompts — suitable, not perfect). Students w32/L2, EXACTLY 600 updates, batch 32, identical inits per seed (asserted), checkpoint by oracle-val hard CE.

Teacher suitability was audited before freezing (§15): w32/L2, 800 updates, validation loss 0.067–0.072 across seeds, greedy fully-valid traces on 87/100 train prompts, tokenizer compatibility gate passed, scoring is a single forward pass per prefix (no generation needed, so the Ch18 free-generation weakness does not disqualify it). Arms: A fixed teacher-greedy rows (799/800/800 after counted structural exclusion), B fresh student samples, C stale v0 buffer. Objective everywhere: conditional forward KL, T=1.0, detached sampling. Test ledger appended; nothing tuned on test outcomes.

Offline versus on-policy results

Example F, the real comparison (per-seed ID exact):

Seed A-offline B-on-policy C-stale
10 0.343 0.365 0.065
11 0.488 0.477 0.103
12 0.528 0.515 0.060

Aggregates: A 0.453 ± 0.097, B 0.452 ± 0.078 (gap −0.001), C 0.076 ± 0.024. OOD: A 0.050, B 0.051, C 0.033. Trace validity: A 0.433, B 0.440, C 0.020. Forced-step: A 0.950, B 0.952, C 0.846. The primary hypothesis is not supported: fresh student prefixes do not improve held-out correctness over fixed teacher prefixes here. The secondary staleness hypothesis is consistent with the result: C collapses. But C replayed samples from the untrained initial student, which were stale and near-garbage at once, so the experiment does not separate staleness from sample quality. Prefix source was neutral when fresh; stale replay of those samples was destructive.

Fresh prefixes tie; stale prefixes collapse

Figure 3. Held-out ID exact for the three arms: A and B tie to three decimals while C loses 0.38 points. Frozen E3 measurements.

Actual optimiser and token budgets

All 9 students ran exactly 600 updates; inits identical within seed (asserted, not assumed). Target-token exposure: A ~325k, B ~317k, C 289–334k per seed — near-equal, so volume confounds nothing. Teacher forwards: A ~13.5k batched (dataset build, seconds of wall time) versus B/C 600 queries each — on-policy uses fewer teacher forwards here, because one greedy teacher decode costs a forward per token while one training step costs one scorer forward. Wall time reverses the ratio: B 134–151 s/arm versus A 84–89 s (~1.6×), because sampling is sequential while dataset decoding is batched. Report both ledgers; either alone misleads.

The marginal arithmetic is stark: on-policy training spends ~50 extra wall seconds per arm (plus 600 sequential student forwards per update in sampling overhead) for a measured ID gap of −0.001 ± ~0.02 — no benefit per unit compute on this task. Against that, the stale control shows what the sampling machinery is worth when freshness and sample quality are removed together: −0.38 ID points for saving nothing (C costs the same 600 updates). The economics lesson is therefore conditional: freshness is cheap insurance against a large loss — one demonstrated here only for replayed initial-student samples, where staleness and sample quality are confounded — not a purchased gain.

Distribution-shift measurements

B’s format_ok rate rises from 0.27–0.39 (first 50 updates) to 0.984–0.988 (last 50): the student genuinely learns to visit valid states. Distinct sampled completions narrow from ~0.94 early to ~0.80–0.88 late — mild concentration, no collapse. Yet greedy correctness never exceeds the offline arm. This is the book’s recurring lesson in new clothes: improved coverage of the state space is not improved capability at the evaluated states. The reverse-KL probe (eval-only, labelled diagnostic) agrees: A 0.158, B 0.160, C 1.097 — fresh on-policy training leaves the student/teacher gap unchanged while stale replay blows it open.

Sampling variance

Example E. For IID estimates with E[X_i] = μ and Var(X_i) = σ², linearity of expectation gives E[X̄] = μ, and independence gives \(Var(\bar X) = (1/n^2)\sum Var(X_i) = \sigma^2/n\): the standard error shrinks as 1/√n. But on-policy samples are neither independent (shared evolving policy) nor identically distributed (the policy changes every step), so the formula is a reference, not a licence. Empirically: per-seed B−A gaps are +0.022, −0.011, −0.013 — sign-inconsistent with spread ~0.02, the size of the seed noise itself.

With three seeds, “B ties A” means the gap is indistinguishable from sampling variation, not that the true gap is exactly zero. No variance-reduction claim is made; the design buys matched budgets and identical inits instead, which is what makes the tie interpretable.

Student diversity and mode collapse

The T=1.0 five-sample probe on 32 test prompts: distinct ratios A 0.59–0.75, B 0.68–0.74, C 0.94–0.98 — the collapsed arm is the most “diverse”, because random garbage rarely repeats. Diversity without correctness is noise; the chapter refuses the equation of the two. Invalid-output rate is 0.0 everywhere (all samples terminate). No mode collapse is observed in B: late-training distinctness ~0.85 with correctness intact. Had collapse appeared only as a fixture, it would be labelled as such; here the measurement is on real samples and shows that the reverse-KL-collapse worry did not materialise here under a forward-KL surrogate, as the direction analysis predicts.

Failure modes and negative findings

The headline negative stands: on-policy prefix matching did not improve correctness. Plausible contributors, each stated as hypothesis, not finding: the teacher is already ~87% valid, so offline prefixes are high-quality and the student’s error states are mild; the forward-KL surrogate cannot exploit distribution information beyond per-token conditionals; 600 updates may be too few for compounding-error regimes to dominate. The stale arm is the positive control that the setup can detect a combined effect of staleness and sample quality: C loses 0.38 ID points. Reward hacking (MiniLLM §3.4, P-015) was not observed — no degenerated high-teacher-score strings appeared in audited samples — which is consistent with a forward-KL surrogate rather than a policy-gradient objective, and is reported as an observation with that scope, not a general claim.

One genuine pipeline failure occurred before the frozen run and is preserved here: the smoke configuration (4-update teacher) generated zero structurally valid offline rows, which would have left Arm A empty. The runner has no fallback — an empty offline set aborts instead of relaxing the filter — so the smoke used a 50-update teacher purely as a pipeline check, and the frozen run’s universal exclusion (1/0/0 rows counted, never hidden) is the same rule operating as designed.

On-policy distillation versus reinforcement learning

Two different optimised objects. On-policy distillation (this chapter, GKD with λ=1 and forward KL): sample from the student, minimise a divergence to fixed teacher conditionals, no gradient through sampling, no reward, no baseline, no trust region. MiniLLM (P-015) does use policy gradients with teacher-mixed sampling, length normalisation, and clipping — closer to RLHF machinery, and the chapter cites it as the contrast that keeps our surrogate honest. GKD + RL (P-016 §3.2) optimises reward plus distillation jointly; nothing here optimises reward at all. Same student samples, different mathematics, different claims. No policy gradients were computed in the making of this chapter — the exact-vs-surrogate test (test_exact_vs_surrogate_gradient_distinction) exists to prove the negative.

Rights, provenance, and limits

Every sampled batch carries student identity, policy version, step, seed, temperature, and cost; replay records keep sampling version and age; the offline set carries a content fingerprint the runner re-verifies before training. RM-112 permits exactly generate_student_trajectories_and_train_on_policy_e3; RM-104 remains pending; nothing is redistributed. Limits, without softening: one synthetic task, three seeds, tiny models, forward-KL surrogate only (no reverse-KL training arm), high-quality offline data (isolates prefix source, not data correctness), OOD as longer chains only, no replication, no independent review.

The strongest caveat is also the most interesting: because A and B tie everywhere measured, this study cannot tell us when on-policy wins — only that freshness without correctness gains is possible, and that stale replay of initial-student samples was destructive here, with staleness and sample quality unseparated.

Preparing for self-distillation

Filtering fixed datasets (Chapter 19) is the end of the off-policy road; matching the prefix distribution (this chapter) narrows the gap in the sense of valid-state visitation — and correctness still did not move. The lever that remains is the supervision source itself: Chapter 21 asks what happens when the teacher is a version of the student. It inherits the sampler (for student-generated data), the scorer pattern (teacher replaced by prior self), the provenance records (versioning becomes generation tracking), and the warning this chapter earned: matching the distribution is not matching the capability, and only held-out correctness can tell the difference.

References

Gu, Y., Dong, L., Wei, F. and Huang, M. (2024). MiniLLM: Knowledge Distillation of Large Language Models. ICLR 2024. https://arxiv.org/abs/2306.08543

Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos, S., Geist, M. and Bachem, O. (2024). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024. https://arxiv.org/abs/2306.13649

Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., et al. (2025). Qwen3 Technical Report. arXiv:2505.09388. https://arxiv.org/abs/2505.09388