On-Policy Distillation
Train the student on its own generated trajectories with teacher supervision: forward and reverse KL, the distribution-mismatch argument, detached-prefix gradient surrogates, and a controlled E3 comparison of offline, on-policy, and stale-replay distillation.
Chapters 17 through 19 trained the student on fixed collections: oracle answers, oracle traces, teacher-generated candidates, verifier-selected subsets. The training states never changed, no matter how the student changed. This chapter asks what happens when they do:
Does supervising the student on its own generated outputs improve the match between training and inference distributions, and at what cost?
The honest answer, on our task, is two-sided. Matching the prefix distribution does nothing measurable for correctness — the on-policy student ties the offline student to three decimals (0.452 versus 0.453). But the choice is not irrelevant: training on samples drawn once from the untrained initial student and replayed throughout collapses correctness from 0.45 to 0.08. Those samples were both stale and near-garbage, so the experiment cannot say which of the two did the damage. Where the states come from matters enormously when they are wrong and stale, and not measurably, here, when they are fresh. That asymmetry is the chapter’s finding, and the machinery below exists to make it checkable rather than merely plausible.
Why offline distillation misses student errors
A student trained by teacher forcing learns conditional distributions: given a reference prefix, predict the next token. At inference it conditions on its own prefixes, which may wander into states the reference data never visits. An early mistake compounds — each subsequent prediction conditions on the mistake — which is the mechanism behind exposure bias (Bengio et al., 2015; Arora et al., 2022, P-016 register). Offline distillation supervises exactly the states of a fixed dataset; it never sees the student’s own error states, so it cannot correct them. On-policy distillation (P-015, P-016) samples from the student, scores those states with the teacher, and trains there. Whether that helps is empirical; that it changes where training happens is definitional.
KD-024 — On-policy distillation
On-policy distillation (KD-024): student-generated trajectories are supervised by a teacher, reducing the mismatch between training and inference distributions. The defining property is the source and freshness of the sampled states: prefixes drawn from the current student policy, scored by the teacher, used within the same training window. A dataset generated once by an older checkpoint and reused for hundreds of updates is not on-policy relative to a changing student — it is replay, and Example D below shows why the distinction matters.
KD-025 — Off-policy distillation
Off-policy distillation (KD-025): a fixed teacher-generated dataset is used independently of the student’s current policy. Teacher traces (Chapter 18), verifier-selected candidates (Chapter 19), and ground-truth answers (Chapter 17) are all off-policy in this sense: the behaviour policy that produced the states — oracle, teacher, or verifier filter — is not the student being trained. Record the actual behaviour policy wherever known; “offline” alone does not say whose outputs the student sees.
Which model generates the training states?
Two pipelines, different state sources. Off-policy: a fixed behaviour policy generates data; data is stored; the student trains against the fixed distribution. On-policy: the current student generates prefixes; the teacher scores those states; the student updates; the sampling distribution evolves with the student. The reader must keep one fact in view: the key difference is where training states come from, not whether a teacher is present. Both pipelines in this chapter use the same frozen teacher and the same conditional loss. Only the prefix source differs — which is exactly what makes the comparison a test of KD-024 rather than a test of everything at once.
flowchart LR
F[fixed behaviour policy] --> D[stored dataset]
D --> S1[student trains on fixed states]
S2[student samples own prefixes] --> T[teacher scores those states]
T --> U[student updates; distribution evolves]
Figure 1. Off-policy (top) reuses one fixed distribution; on-policy (bottom) trains where the student currently stands. Same teacher, same conditional loss — only the state source differs.
Forward KL
Let p_T(y|x) be the teacher distribution and p_S(y|x) the student distribution over outputs y given input x. The forward KL is:
\(D_{KL}(p_T \mid\mid p_S) = \sum_y p_T(y|x) \log \frac{p_T(y|x)}{p_S(y|x)}\)
The expectation is teacher-weighted: every output counts in proportion to its teacher probability. Where the teacher assigns mass, the student must also assign mass or pay steeply — forward KL is zero-avoiding on the student side, which is why it is mode-covering: a student that cannot represent all teacher modes stretches across them, assigning unreasonable probability to regions the teacher rates near zero (Huszár, 2015; MiniLLM §2, P-015). It corresponds, under an empirical data distribution, to maximum likelihood — the familiar cross-entropy objective is forward KL plus a constant teacher entropy. Support condition: wherever p_T > 0 the student must keep p_S > 0, or the divergence is infinite. Missing a teacher mode is the expensive error.
Reverse KL
\(D_{KL}(p_S \mid\mid p_T) = \sum_y p_S(y|x) \log \frac{p_S(y|x)}{p_T(y|x)}\)
Now the expectation is student-weighted: outputs count in proportion to student probability. Probability the student places where the teacher assigns little is penalised directly — reverse KL is mode-seeking: the student concentrates on teacher modes it can represent and ignores the rest. Support condition: wherever the student puts mass, the teacher must have mass, or the penalty is unbounded. The failure it invites is the mirror image: collapse onto one mode while the task needs several. Do not claim reverse KL always causes mode collapse; it prefers concentration, and whether that hurts depends on whether the ignored modes matter for correctness.
Mode covering and mode seeking
Example A makes the asymmetry concrete. Teacher T = [0.7, 0.2, 0.1], student S = uniform [1/3, 1/3, 1/3]. Forward KL(T||S) = 0.7·ln(2.1) + 0.2·ln(0.6) + 0.1·ln(0.3) ≈ 0.297. Reverse KL(S||T) = (1/3)·ln(0.476)
- (1/3)·ln(1.667) + (1/3)·ln(3.333) ≈ 0.324. Different numbers, different incentives: the forward direction charges 0.7·ln(2.1) for under-covering the teacher’s main mode; the reverse direction charges (1/3)·ln(3.333) for the student’s mass on the teacher’s weakest outcome.
On open-ended generation with limited student capacity, the forward penalty forces coverage of long-tail teacher variants the student will then emit at inference (the MiniLLM §1 argument, P-015); the reverse penalty forbids exactly those emissions at the price of neglected modes. Neither direction is universally correct; each buys one failure mode and sells the other.
Figure 2. Example A drawn as bars: the same teacher-student pair scores 0.297 forward and 0.324 reverse, charging different mistakes. Worked arithmetic, test-pinned, not measured data.
Autoregressive sequence probabilities
For an output sequence y = (y_1 … y_L):
\(p(y|x) = \prod_{t=1}^{L} p(y_t \mid x, y_{ Each factor is a token-level conditional on the prompt plus the
preceding prefix. Teacher forcing trains these conditionals on
reference prefixes; free generation evaluates their composition on
self-produced prefixes. Masks and denominators must be explicit: loss
averages over supervised positions only (completion span, never PAD),
and the sequence log-probability is the sum of conditional log
probabilities — a fact the tests verify by both routes (test_autoregressive_factorisation).
Variable length is handled by EOS: positions after EOS contribute
nothing, and rows carry PAD to a fixed length that masking excludes. Sequence-level forward KL weights prefixes by the teacher
distribution; sequence-level reverse KL weights them by the student
distribution. That weighting is the connection to state visitation:
forward-KL training supervises teacher-visited states, reverse-KL
training supervises student-visited states. The decomposition is
exact. Write sequence forward KL by factoring the first token: \(D_{KL}(p_T(y|x) \mid\mid p_S(y|x)) = E_{y_1 \sim p_T}[\log p_T(y_1|x)/p_S(y_1|x)] + E_{y_1 \sim p_T}[D_{KL}(p_T(\cdot|x,y_1) \mid\mid p_S(\cdot|x,y_1))]\) and recurse: at every position, the conditional divergence is averaged
over prefixes drawn from the teacher’s prefix distribution. Swapping
the roles of p_T and p_S throughout — student prefixes outside and the
reverse conditional KL(p_S‖p_T) inside — yields sequence reverse KL.
Swapping only the outer sampler, while keeping the forward conditional,
does not. Two consequences
follow. First, teacher-prefix and student-prefix supervision can
behave differently even with identical conditional losses — the same
per-token divergence, averaged over different prefix distributions, is
a different objective. Second, our surrogate (conditional forward KL
at detached student prefixes) shares only the outer weighting of the
reverse decomposition — student prefixes — while keeping the forward
conditional inside and dropping the gradient through sampling — precisely
the gap the six-point documentation rule below labels. Example
B: a teacher trace for 7 + 4 then × 3 reads STEP 7+4=11, STEP
11×3=33. A student that instead generates STEP 7+4=12 will, under
offline training, never be supervised at the state “12 × 3” — yet at
inference that is exactly where it stands. On-policy training puts the
teacher’s conditional (what follows “12 × 3”?) at the student’s
actually-visited prefix. Whether the student benefits is the
experiment, not the derivation. This is the chapter’s critical technical point. Suppose the student
generates prefixes and we compute token-level teacher/student KL at
those prefixes, treating the sampled prefixes as detached
observations. The gradient then does not account for how the prefix
distribution changes with student parameters — the sampling step is
stopped-gradient. That implementation is not the exact gradient of
sequence-level reverse KL. In particular: student-generated prefixes
combined with conditional forward KL are not exact sequence-level
reverse-KL optimisation. Exact reverse-KL gradients need the
score-function term through the sampling distribution (MiniLLM Eq. 2,
P-015); GKD-style detached conditionals (P-016, Eq. 4) deliberately
omit it for stability. Document separately: (1) which policy produces
prefixes, (2) which conditional divergence direction, (3) which
parameters receive gradients, (4) whether sampling gradients are
included, (5) the actual estimator, (6) the approximated objective. An
accurately labelled stop-gradient conditional-KL surrogate is
acceptable for the bounded experiment. Our primary objective is
therefore stated plainly: an on-policy prefix-distribution surrogate
using conditional forward KL — student-sampled prefixes, forward
conditional direction, gradients through student logits only. Example C is the actual training step (abridged; the runner adds
accounting): Sampling must come from the current parameters — the sampler runs in
For each sampled prefix the frozen teacher supplies full conditional
logits — not just an answer label. The teacher runs under Example D: at step 0 the student is randomly initialised, so its
samples are near-garbage token strings; at step 500 the same prompts
elicit structured traces. A buffer frozen at version 0 and reused at
step 500 supervises states the student no longer visits, with a
staleness age of 500. Arm C is exactly this: one sample per training
prompt from the initial student, resampled in batches, age growing 0
to 600. GKD’s λ parameter (P-016) spans this spectrum deliberately —
λ=1 pure on-policy, λ=0 supervised, mixtures between. Our C is the
λ=0-in-spirit extreme with student-generated (not teacher) fixed data,
which confounds staleness with sample quality, because the frozen samples
come from the untrained student. Run Teacher suitability
was audited before freezing (§15): w32/L2, 800 updates, validation
loss 0.067–0.072 across seeds, greedy fully-valid traces on 87/100
train prompts, tokenizer compatibility gate passed, scoring is a
single forward pass per prefix (no generation needed, so the Ch18
free-generation weakness does not disqualify it). Arms: A fixed
teacher-greedy rows (799/800/800 after counted structural exclusion),
B fresh student samples, C stale v0 buffer. Objective everywhere:
conditional forward KL, T=1.0, detached sampling. Test ledger
appended; nothing tuned on test outcomes. Example F, the real comparison (per-seed ID exact): Aggregates: A 0.453 ± 0.097, B 0.452 ± 0.078 (gap −0.001), C 0.076 ±
0.024. OOD: A 0.050, B 0.051, C 0.033. Trace validity: A 0.433, B
0.440, C 0.020. Forced-step: A 0.950, B 0.952, C 0.846. The primary
hypothesis is not supported: fresh student prefixes do not improve
held-out correctness over fixed teacher prefixes here. The secondary
staleness hypothesis is consistent with the result: C collapses. But C
replayed samples from the untrained initial student, which were stale and
near-garbage at once, so the experiment does not separate staleness from
sample quality. Prefix source was neutral when fresh; stale replay of
those samples was destructive. Figure 3. Held-out ID exact for the three arms: A and B tie to three
decimals while C loses 0.38 points. Frozen E3 measurements. All 9 students ran exactly 600 updates; inits identical within seed
(asserted, not assumed). Target-token exposure: A ~325k, B ~317k, C
289–334k per seed — near-equal, so volume confounds nothing. Teacher
forwards: A ~13.5k batched (dataset build, seconds of wall time) versus
B/C 600 queries each — on-policy uses fewer teacher forwards here,
because one greedy teacher decode costs a forward per token while one
training step costs one scorer forward. Wall time reverses the ratio:
B 134–151 s/arm versus A 84–89 s (~1.6×), because sampling is
sequential while dataset decoding is batched. Report both ledgers;
either alone misleads. The marginal arithmetic is stark: on-policy
training spends ~50 extra wall seconds per arm (plus 600 sequential
student forwards per update in sampling overhead) for a measured ID
gap of −0.001 ± ~0.02 — no benefit per unit compute on this task.
Against that, the stale control shows what the sampling machinery is
worth when freshness and sample quality are removed together: −0.38 ID points for saving nothing
(C costs the same 600 updates). The economics lesson is therefore
conditional: freshness is cheap insurance against a large loss — one
demonstrated here only for replayed initial-student samples, where
staleness and sample quality are confounded — not a purchased gain. B’s format_ok rate rises from 0.27–0.39 (first 50 updates) to
0.984–0.988 (last 50): the student genuinely learns to visit valid
states. Distinct sampled completions narrow from ~0.94 early to
~0.80–0.88 late — mild concentration, no collapse. Yet greedy
correctness never exceeds the offline arm. This is the book’s
recurring lesson in new clothes: improved coverage of the state space
is not improved capability at the evaluated states. The reverse-KL
probe (eval-only, labelled diagnostic) agrees: A 0.158, B 0.160, C
1.097 — fresh on-policy training leaves the student/teacher gap
unchanged while stale replay blows it open. Example E. For IID estimates with E[X_i] = μ and Var(X_i) = σ²,
linearity of expectation gives E[X̄] = μ, and independence gives
\(Var(\bar X) = (1/n^2)\sum Var(X_i) = \sigma^2/n\): the standard error
shrinks as 1/√n. But on-policy samples are neither independent (shared evolving
policy) nor identically distributed (the policy changes every step),
so the formula is a reference, not a licence. Empirically: per-seed
B−A gaps are +0.022, −0.011, −0.013 — sign-inconsistent with spread
~0.02, the size of the seed noise itself. With three seeds, “B ties A” means the gap is
indistinguishable from sampling variation, not that the true gap is
exactly zero. No variance-reduction claim is made; the design buys
matched budgets and identical inits instead, which is what makes the
tie interpretable. The T=1.0 five-sample probe on 32 test prompts: distinct ratios A
0.59–0.75, B 0.68–0.74, C 0.94–0.98 — the collapsed arm is the most
“diverse”, because random garbage rarely repeats. Diversity without
correctness is noise; the chapter refuses the equation of the two.
Invalid-output rate is 0.0 everywhere (all samples terminate). No
mode collapse is observed in B: late-training distinctness ~0.85 with
correctness intact. Had collapse appeared only as a fixture, it would
be labelled as such; here the measurement is on real samples and
shows that the reverse-KL-collapse worry did not materialise here under
a forward-KL surrogate, as the direction analysis predicts. The headline negative stands: on-policy prefix matching did not
improve correctness. Plausible contributors, each stated as
hypothesis, not finding: the teacher is already ~87% valid, so
offline prefixes are high-quality and the student’s error states are
mild; the forward-KL surrogate cannot exploit distribution information
beyond per-token conditionals; 600 updates may be too few for
compounding-error regimes to dominate. The stale arm is the positive
control that the setup can detect a combined effect of staleness and
sample quality: C loses 0.38 ID points. Reward hacking (MiniLLM §3.4, P-015) was not
observed — no degenerated high-teacher-score strings appeared in
audited samples — which is consistent with a forward-KL surrogate
rather than a policy-gradient objective, and is reported as an
observation with that scope, not a general claim. One genuine
pipeline failure occurred before the frozen run and is preserved
here: the smoke configuration (4-update teacher) generated zero
structurally valid offline rows, which would have left Arm A empty.
The runner has no fallback — an empty offline set aborts instead of
relaxing the filter — so the smoke used a 50-update teacher purely
as a pipeline check, and the frozen run’s universal exclusion (1/0/0
rows counted, never hidden) is the same rule operating as designed. Two different optimised objects. On-policy distillation (this
chapter, GKD with λ=1 and forward KL): sample from the student,
minimise a divergence to fixed teacher conditionals, no gradient
through sampling, no reward, no baseline, no trust region. MiniLLM
(P-015) does use policy gradients with teacher-mixed sampling,
length normalisation, and clipping — closer to RLHF machinery, and
the chapter cites it as the contrast that keeps our surrogate honest.
GKD + RL (P-016 §3.2) optimises reward plus distillation jointly;
nothing here optimises reward at all. Same student samples, different
mathematics, different claims. No policy gradients were computed in
the making of this chapter — the exact-vs-surrogate test (test_exact_vs_surrogate_gradient_distinction)
exists to prove the negative. Every sampled batch carries student identity, policy version, step,
seed, temperature, and cost; replay records keep sampling version and
age; the offline set carries a content fingerprint the runner
re-verifies before training. RM-112 permits exactly
The strongest caveat is also the
most interesting: because A and B tie everywhere measured, this study
cannot tell us when on-policy wins — only that freshness without
correctness gains is possible, and that stale replay of initial-student
samples was destructive here, with staleness and sample quality
unseparated. Filtering fixed datasets (Chapter 19) is the end of the off-policy
road; matching the prefix distribution (this chapter) narrows the gap
in the sense of valid-state visitation — and correctness still did not
move.
The lever that remains is the supervision source itself: Chapter 21
asks what happens when the teacher is a version of the student.
It inherits the sampler (for student-generated data), the scorer
pattern (teacher replaced by prior self), the provenance records
(versioning becomes generation tracking), and the warning this
chapter earned: matching the distribution is not matching the
capability, and only held-out correctness can tell the difference. Gu, Y., Dong, L., Wei, F. and Huang, M. (2024). MiniLLM: Knowledge
Distillation of Large Language Models. ICLR 2024.
https://arxiv.org/abs/2306.08543 Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos, S., Geist,
M. and Bachem, O. (2024). On-Policy Distillation of Language Models:
Learning from Self-Generated Mistakes. ICLR 2024.
https://arxiv.org/abs/2306.13649 Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., et al.
(2025). Qwen3 Technical Report. arXiv:2505.09388.
https://arxiv.org/abs/2505.09388Teacher prefixes versus student prefixes
Exact objectives and practical gradient surrogates
Building the on-policy loop
examples/instruction_distillation/on_policy.py implements the
minimal loop on the shared workbench: sample continuations from the
current student (temperature multinomial, seeded per step, EOS and
length enforced), assemble prompt+sample rows with a sampled-span mask,
score with the frozen teacher (TeacherScorer, query/token ledger),
compute conditional forward KL on the span, backpropagate through the
student only, clip, step, and checkpoint by hard CE on the oracle
validation split after the full budget. Replay buffer stores rows with
their sampling version; only version == current step counts as fresh.
Failures are explicit: empty samples, over-length contexts, and
non-finite losses raise instead of silently continuing.Sampling from the current student
gen = torch.Generator().manual_seed(seed * 1_000_000 + step)
idx = torch.randint(0, n_train, (32,), generator=gen).tolist()
prows = [prompts[i] for i in idx]
conts, ledger = on_policy.sample_continuations(
student, torch.tensor(prows), max_new=32, temperature=1.0,
generator=gen, eos_id=rsn.EOS_ID, max_length=64)
rows, mask = on_policy.rows_from_samples(prows, conts, pad_id=0, row_length=64)
opt.zero_grad(set_to_none=True)
s_logits = student(rows[:, :-1])
t_logits = scorer.score(rows[:, :-1], mask=mask[:, 1:])
loss = losses.token_kl(s_logits, t_logits, rows[:, 1:], temperature=1.0, mask=mask[:, 1:])
on_policy.check_finite(loss, what="student loss")
loss.backward()
eval/no-grad mode on the live student object, and the provenance
record carries policy version, step, seed, and temperature. Reusing an
old checkpoint’s outputs here would be replay mislabelled as
on-policy; the buffer’s version field exists to make that fraud
structurally visible.Teacher supervision on student states
no-grad,
is switched to eval once, and a test asserts its parameters are
bit-identical before and after scoring while student gradients are
nonzero and no teacher gradient exists. Masking covers sampled
positions only: prompts are given, not learned here. If a sample
cannot be scored (empty continuation, over-length context), the run
raises an explicit error rather than fabricating probabilities. No
future-token leakage: teacher and student see identical shifted
contexts, and the mask aligns prediction positions exactly.Stale trajectories and replay
The frozen E3 experiment
E3O-20261116T073939Z-3dbca7, protocol 1.0, rights RM-112
(generate_student_trajectories_and_train_on_policy_e3,
verified-permitted, gate first). Fresh deterministic splits, seeds 10,
11, 12: train 800, val 200, test 600, ood 400; 2-step train, 3-step
OOD. Teacher w32/L2, 800 updates, oracle trace+answer mask (greedy
fully-valid 87/100 on train prompts — suitable, not perfect).
Students w32/L2, EXACTLY 600 updates, batch 32, identical inits per
seed (asserted), checkpoint by oracle-val hard CE.Offline versus on-policy results
Seed
A-offline
B-on-policy
C-stale
10
0.343
0.365
0.065
11
0.488
0.477
0.103
12
0.528
0.515
0.060
Actual optimiser and token budgets
Distribution-shift measurements
Sampling variance
Student diversity and mode collapse
Failure modes and negative findings
On-policy distillation versus reinforcement learning
Rights, provenance, and limits
generate_student_trajectories_and_train_on_policy_e3; RM-104 remains
pending; nothing is redistributed. Limits, without softening: one
synthetic task, three seeds, tiny models, forward-KL surrogate only
(no reverse-KL training arm), high-quality offline data (isolates
prefix source, not data correctness), OOD as longer chains only, no
replication, no independent review.Preparing for self-distillation
References