Distilling Reasoning
Do worked procedures teach reasoning, or only its format? This chapter builds a checkable arithmetic trace task, compares answer-only against trace supervision with matched update budgets, and measures what transfers to longer problems.
Trace-trained students in this chapter’s experiment predicted the next token of a worked arithmetic trace correctly about 90% of the time when they were fed the correct trace so far. Left to write the trace themselves, they produced a fully valid one only 2–6% of the time. Knowing each step when shown the way there is not the same as being able to walk it, and a reasoning claim has to rest on the second.
Chapter 17 distilled answers. This chapter asks whether supervising the working changes what the student learns:
Can a student learn to reason from a teacher’s traces, and how would we know that it is reasoning rather than copying an answer format?
The question splits in two on contact. First, can trace supervision teach at all here — do students that see worked procedures outperform students that see only answers? Second, when traces help or fail to help, is the mechanism reasoning or formatting — and what measurement could possibly tell? The chapter’s answer to the first is a qualified, task-specific maybe-not (answers win in-distribution); its answer to the second is machinery: a step checker, a chaining rule, a longer-chain held-out set, and a token-matched control, assembled so that the most comforting wrong answers are unavailable.
The answer this chapter defends is sceptical and specific: on a small task with checkable steps, trace supervision teaches the format of reasoning fluently and the arithmetic barely at all. Students that cannot add still write beautiful proofs of wrong answers — and the apparatus below catches them red-handed, per step, per seed. The evidence is a controlled four-arm ablation with a held-out longer-chain test — and a set of validity checks that refuse to mistake form for thought.
Answers are not reasoning
A correct answer with a convincing explanation is not necessarily valid reasoning. Consider a student that writes, for 7 + 4 then × 3:
STEP 7 + 4 = 11, STEP 11 × 3 = 33, answer 33.
Every number follows from the last; the answer is right; the trace is a proof. Now consider a student that writes:
STEP 7 + 4 = 12, STEP 12 × 3 = 36, answer 36.
Same format, same confidence, wrong at the first step — and the wrong answer follows validly from the wrong step. And a third student:
STEP 7 + 4 = 12, STEP 12 × 3 = 33, answer 33.
Right answer, broken middle: the steps do not chain, but the conclusion happens to match. An accuracy metric scores the first and third identically. A format check passes all three. Only a step-by-step oracle distinguishes them — and this chapter builds exactly that oracle, then shows how much of the apparent reasoning survives it. Spoiler: the format survives; the arithmetic mostly does not.
Figure 1. The opening trio as checkable cases: only step-by-step re-execution tells the proof from the lucky guess. Illustration of the worked example, not measured data.
KD-022 and the reasoning question
Reasoning distillation (KD-022) is the transfer of intermediate reasoning behaviour or process supervision, not only final answers. The registry marks it Emerging: task-dependent, with thin evidence — a status this chapter earns rather than merely cites. Behavioural means testable: a reusable procedure must work on new combinations and, in our OOD design, longer chains. A student that solves only trained lengths has memorised templates; one that extends them has acquired something procedure-like. The bar is deliberately behavioural rather than architectural: no probe of weights or attention maps could substitute for held-out problems the student has never seen. Two confusions must be refused upfront.
First, a trace is not a thought: our worked procedures are deterministic oracle outputs, explicitly not hidden reasoning inside a language model, and nothing here claims access to any model’s private cognition. Second, imitation is not acquisition: a student that copies the STEP … = … template without computing has learned formatting, and formatting is cheap. The chapter’s entire evaluation apparatus — step validity, chaining checks, longer-chain held-out tests — exists to separate the two.
A task with checkable steps
Chain arithmetic: a starting value, a sequence of +/−/× operations on
small operands, and an oracle that executes them. The worked example
above is real training data: problem [7, +4, ×3], values [7, 11, 33], encoded as digit tokens with STEP/ANS/EQ markers and a
terminating EOS (27 non-pad tokens for 2 steps). Every intermediate
value is recomputable by hand, which makes each step independently
checkable — the property a reasoning task needs and most benchmarks
lack. Validity is not a vibe; it is apply_op re-executed per step.
Three design choices matter. First, values stay small: v0 in 1..9, operands 1..6 (×2..3), two chained steps, results bounded in practice below 100 — every number is one or two digit-tokens, so the trace length measures reasoning steps, not numeral width. Second, splits are by exact problem identity (inputs, ops, operands): 800/200/600 per seed for train/val/ID-test, plus 400 longer 3-step problems as OOD that no training row resembles in length. The ID test uses new combinations at the trained length (can it recombine?); OOD uses a new length (can it extend?). Leakage is checked, not assumed: zero overlapping problems between train and either test split, verified in code, with the check itself tested.
Third, the task is deliberately too small to be interesting and exactly small enough to be decisive: if trace supervision cannot demonstrate an advantage here, with perfect step labels and a tiny space, the burden of proof is on anyone claiming it works at scale — and, as the results show, our own experiment does not lift it either.
A concrete row makes the
abstraction tangible. Problem: start 7, add 4, multiply by 3 — values
[7, 11, 33]. As tokens: [BOS, PROBLEM, 7, +, 4, EOS | STEP, 7, +, 4, =, 11 | STEP, 11, *, 3, =, 33 | ANS, 33, EOS] (27 non-pad tokens for
2 steps; digits, signs and markers are single ids each). The prompt
ends at the first EOS; the trace is the two STEP blocks; the answer is
ANS plus digits plus final EOS. Change any single digit and exactly one
validity bit flips — the representation is as brittle as the reasoning
it records, which is why the checker, not the eye, judges it.
Traces, validity, and the chaining rule
A trace record carries: problem identity (the exact op/operand sequence), input, reference answer, per-step records (input value, op, operand, result), length in steps and tokens, and generator provenance (oracle procedure vs trained teacher, with decoding configuration where applicable). Step types are uniform here — every step is an arithmetic transition — which is what makes the checker total: there is no step kind it cannot re-execute. Validity decomposes into four independent booleans:
format_ok: markers present and ordered (STEPs, ANS, EOS).step_valid[i]: step i’s arithmetic recomputes.chain_ok: the trace starts from the prompt’s problem and each step consumes the previous step’s result.final_ok: the answer field matches the trace’s own last result.
The six combinations the contract demands are all constructible as
fixtures and all verified by the checker: valid/valid; invalid step
with correct final (the running example — first step wrong, answer
accidentally right); valid trace with wrong answer (answer field
corrupted); truncated traces (missing ANS); template copying (a valid
trace from problem B transplanted onto problem A — internally
consistent, chain_ok false, the precise failure the chaining rule
exists to catch); and unverifiable malformed rows. The template-copy
case deserves emphasis: without chain_ok, a copied trace passes every
local check. Chaining is what makes a trace about its problem.
Objectives: answers versus traces
Let \(a\) be the answer tokens and \(r\) the trace tokens for input \(x\). Answer-only supervision minimises \(L_A = -\log p(a|x)\); trace-then-answer minimises \(L_{R,A} = -\log p(r,a|x)\); the mixed arm uses \(0.5\,L_{R,A} + 0.5\,L_A\) (answer upweighted, predeclared). All losses are completion-masked cross-entropies; the mask is what makes the same row supervise different things per arm. Formally, with mask \(m\) selecting supervised positions and mean reduction:
\[ L = -\frac{\sum_t m_t \log p(y_t \mid y_{<t}, x)}{\sum_t m_t}, \]
so each supervised token contributes equally and each unsupervised token contributes nothing. Change the mask and you change the task; the optimiser never sees the difference.
Trace-length bias is structural, not incidental. Answer spans average ~3 tokens; trace-plus-answer spans average ~23. Under a mean reduction, each token counts equally, so in the mixed arm the trace commands roughly 23/(23+3) ≈ 88% of the gradient before the explicit 0.5/0.5 weights even apply — and with a sum reduction it would dominate absolutely (the fixture suite asserts the measured ratio on real training rows: ~470k B-tokens vs ~72k A-tokens per seed at matched updates).
The chapter reports the ratio because any “trace supervision helps/hurts” claim is confounded with “the model spent its gradient on 8× more tokens” unless token exposure is measured. It is: every arm logs tokens alongside updates, and the fourth arm (A2, answer-only with an 8× batch) exists to test whether tokens, not traces, drive any difference. A correct answer proves nothing about any intermediate step — and a low trace loss proves nothing about generation, as the results will show.
Failure demonstrations, in fixture form
Six fixtures, each with independently checked expectations — and one
of them deserves its token sequence printed in full, because abstract
failure taxonomies are forgettable and token strings stick. Take the
running example (7, +4, ×3 → 11, 33). Corrupt the first result digit:
[..., STEP, 7, +4, =, 12, ...] where 12 should be 11. The checker
reports step_valid = [False, True] (the second step verifies locally:
something + something = something can be internally consistent while
wrong), chain_ok = False, and — the punchline — final_ok = True
if the answer field still says 33.
A correct answer with a
demonstrably invalid first step, produced by a one-token corruption,
caught only because validity is checked per step rather than per
answer. The remaining fixtures: (2) a correct trace with a
wrong answer field — trace_ok true, final_ok false, the case that
forces answer and process metrics apart; (3) truncation before
ANS — format_ok false with the reason recorded; (4) template copying
across problems — chain_ok false while every local step verifies;
(5) train/test leakage probes — zero overlap asserted on every split;
(6) the length audit — trace tokens outnumber answer tokens ~7:1 on
training rows, so any mean-reduced mixed loss is a trace loss with an
answer footnote. Each fixture is a test, not an anecdote: the suite
asserts the checker’s verdict on all six.
The frozen ablation
Four arms on one shared population per dataset seed (0, 1, 2):
| Arm | Supervision | Batches |
|---|---|---|
| A-answer | oracle answer spans only | 32 |
| B-trace | oracle worked procedures (trace+answer mask) | 32 |
| C-mixed | 0.5·CE(trace+answer) + 0.5·CE(answer), oracle rows | 32 |
| A2-answer-tokmatched | oracle answers, 8× batch (256) | 256 |
All arms: EXACTLY 600 optimiser updates (the Chapter 15 lesson, now routine), same width-32/layers-2 student architecture as the Ch17 teacher scale, best-validation checkpoint restored after the full budget, identical inits across arms within a seed. The 600-update level follows a measured calibration: a 2-step probe shows learning signal at 600 while 1,500 would breach the 30-minute ceiling once the token-matched arm is included — and the level is identical for every arm, so the comparison stays fair at any level. A2 approximates B’s per-update token exposure at the same update count — approximately by predeclared design (8× batch vs ~7× measured tokens), with actuals reported (B ~470k tokens vs A2 ~287k per seed — approximately matched, not exactly, and the shortfall is disclosed, not rounded away).
A local teacher (width-32, 800 updates) is trained per seed as a
generation diagnostic: greedy trace validity 0.49–0.63 on 2-step
problems (vs 0.000 on the abandoned 3-step pilot), reported, not used —
arm B uses oracle traces by predeclared amendment A1, because a
half-valid teacher would confound supervision format with teacher
quality. Rights: RM-110 (train_and_evaluate_e3_reasoning,
verified-permitted), gate passed first. Test and OOD splits evaluated
once each, logged in the test ledger; no decision followed from them.
What the students learned — and did not
Frozen run (E3R-20261115T202834Z-1765b5), held-out answer exact:
| Arm | ID (2-step) | OOD (3-step) | Trace-valid | Forced-step |
|---|---|---|---|---|
| A-answer | 0.093 | 0.000 | — | 0.023–0.043 |
| B-trace | 0.068 | 0.025 | 0.063 | 0.904–0.911 |
| C-mixed | 0.045 | 0.026 | 0.019 | 0.876–0.894 |
| A2-tokmatched | 0.098 | 0.001 | — | 0.023–0.043 |
(means over 3 seeds; trace-valid and forced-step on ID tests.)
Three findings, all descriptive. First, the forced-vs-free gap: B/C predict trace tokens at ~0.90 teacher-forced but generate valid traces at 0.02–0.06 free-running. The steps are learned and not generatable — exposure bias, measured, not hypothesised. The local teacher shows the same gap, which is why arm B’s oracle supervision was the honest design: free-running multi-step generation defeated these tiny students, and the local teacher managed valid two-step traces only about half the time and three-step traces never.
Note the mixed arm’s particular failure: C (0.045) underperforms B (0.068) on ID on all three seeds despite identical traces. One reading is that upweighting the answer inside a trace objective teaches neither component well with this capacity; the experiment does not isolate that mechanism. Mixing is not averaging; under a fixed budget it can be dilution, and the predeclared weights (0.5/0.5) make that dilution legible rather than hiding it behind a tuned coefficient.
Figure 2. Teacher-forced step accuracy against free-running valid-trace rates for the trace arms, plus the local teacher’s 2-step/3-step diagnostic. E3 reasoning measurements.
Second, answer-only wins ID (A 0.093, A2 0.098 vs B 0.068, C 0.045) while trace supervision whispers on OOD (B/C 0.025–0.026 vs A 0.000 exactly on all seeds). Trace training trades ID accuracy for a non-zero longer-chain tail. With n=400 OOD items and ~10 hits per seed, that tail is suggestive, not established — and the chapter presents it as a direction for follow-up, not a finding.
Third, token volume does not appear to explain it: A2 (4× the tokens of A, batch 256) matches A (0.098 vs 0.093) and beats both trace arms on ID. If supervision volume drove outcomes, A2 would lead everywhere; it leads only where A leads. A2’s exposure still fell short of B’s (about 287k against 470k tokens per seed), so the control is approximate. The reading the evidence favours is format — short generations that stay on the rails versus long generations that derail — rather than volume, but it is a reading, not an isolated cause. Per-seed consistency strengthens the reading: A/A2 beat B/C on ID on all three seeds (A: 0.102/0.092/0.085; B: 0.057/0.072/0.075), so the ordering is not a seed artefact even though the absolute levels are low.
About that OOD whisper: the natural sceptical reading is that longer traces give more lottery tickets — more generated tokens, more chances to stumble onto the answer. Against that reading stands A2: with ~4× A’s tokens it still scores 0.000–0.001 OOD, so raw token count does not buy longer-chain answers. Something about trace training (rather than trace length at inference) may transfer weakly. What exactly is unidentified — step priors, number-shape familiarity, or noise — and the chapter leaves the mechanism open while reporting the asymmetry exactly. The teacher diagnostic completes the picture: trained identically on oracle traces (800 updates), the would-be teacher generates valid 2-step traces only 0.49–0.63 of the time, and 3-step traces never. Generation, not supervision, is the bottleneck at every level of this study.
Do the students reason?
No — not by this chapter’s own standard. A student that reasons would chain steps to new answers; ours generate valid traces at most 6% of the time, copy templates fluently (the format is always perfect — EOS where it belongs, ANS markers in place), and collapse to zero on longer chains except for a whisper. The per-step greedy correctness (B 0.44, C 0.31) against forced-step accuracy of ~0.90 shows partial step knowledge mid-derailment: the model often starts right and loses the thread, exactly what exposure bias predicts. Note the asymmetry the negative controls provide: answer-only arms score ~0.03 on forced trace positions — they never saw traces, and the metric confirms it — so the 0.90 is learned procedure, not background token skill.
What would
change the verdict? Three predeclared bars, none met: (1) greedy
trace validity approaching forced accuracy (generation matches
knowledge); (2) OOD exact match clearly above the answer-only
baseline across seeds (procedure transfers); (3) chain_ok failures
concentrated in arithmetic rather than chaining (the procedure is
right, the sums are wrong — currently both fail together). The OOD
whisper is the one crack in the verdict: 2.5% on longer chains is not
reasoning, but it is also not nothing, and a follow-up with scheduled
sampling or beam search could test whether it grows. The
chapter’s contribution is the apparatus that makes “looks like
reasoning” checkable — and the check, run honestly, mostly fails.
Rights and provenance
Every trace in this study is generated by repository code from a
deterministic oracle — no model, API, or human produced them, and the
provenance record says exactly that, per candidate: generator identity
(oracle procedure v1), the problem’s fingerprint, and the rights
record RM-110. RM-110 (train_and_evaluate_e3_reasoning,
verified-permitted) covers training on those traces and evaluating the
resulting students; the gate passes before any training runs, and a
wrong action is refused (tested). Nothing here is redistributed beyond
the recorded study artifacts, and nothing personal exists in the data
(integer arithmetic has no data subjects). The rights point that
matters for the Part IV arc is forward-looking: oracle traces are
clean because they are synthetic in the strong sense — no teacher,
no terms, no consent questions.
Model-generated traces (Chapter 20’s territory) inherit whatever produced them, and the reader who confuses the two regimes will misjudge both the science and the permissions. A distilled student trained on traces it cannot verify is a liability laundered through formatting; the provenance chain is what lets a downstream user tell the difference. Chapter 19 inherits exactly two things from this chapter, and the inheritance is deliberate: the step checker (which becomes the process verifier — re-execution is already verification, only the name changes) and the failure fixtures (which become the verifier’s test suite — every way a trace can be wrong is a case the verifier must catch). Nothing else carries over: no thresholds, no schedules, no claims. The handoff is machinery, not conclusions.
Limits and the handoff
One synthetic arithmetic task, three seeds, tiny transformers, oracle traces (not learned reasoning), OOD as longer chains only. Three design decisions bound everything above, and the chapter would rather state them than have them discovered. First, oracle traces: nothing here shows a learned teacher’s traces teaching anything — the local teacher cannot generate valid ones, so the transfer question is tested with perfect supervision, which is the kindest possible case for traces and still loses in-distribution. Second, tiny models: a width-32 transformer is where the experiment is affordable, not where reasoning lives; the mechanism (compounding free-running error) should scale, but the verdict (traces lose) might not.
Third, the OOD is longer chains of the same operators: new lengths, not new operations — a harder test (new operators) and an easier one (new combinations, already covered by the ID split) bracket it. The teacher generates valid 2-step traces about half the time and 3-step traces never — reported, not hidden. Three contemporary programmes are explicitly not this experiment, and the chapter records the differences rather than borrowing their prestige. DeepSeek-R1 (2025) trains reasoning behaviour with reinforcement learning and then distils — its programme is RL plus distillation, not trace supervision, and its gains do not transfer to our setting by analogy. LIMO (2025) shows a small curated reasoning set can work after supervised fine-tuning — scoped to its data and models, not a licence to claim small sets suffice generally (our 800-example set suffices only to demonstrate the failure modes above). s1 (2025) distils traces from a proprietary Gemini teacher — a pipeline whose reuse rights for the distilled outputs are not established by its open release, and whose method (budget forcing at inference) is not supervised trace-distillation at all.
No claim about general reasoning, no claim that traces transfer across tasks, no claim that bigger models would show the same pattern (though the mechanism — compounding free-running error — should worry any scale). Chapter 19 inherits the machinery that did work: deterministic step checking, which becomes the process verifier, and the candidate pool’s failure modes, which become its test fixtures. Gate 18 is satisfiable on the evidence above: trace schema frozen, answer and step checking independent, splits clean, three (+1) arms accounted, budgets matched and token-reported, claims limited to measurement, rights passed, tests and rendering green.
References
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. and Zhou, D. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. https://arxiv.org/abs/2201.11903
Magister, L. C., Mallinson, J., Adamek, J., Malmi, E. and Severyn, A. (2023). Teaching Small Language Models to Reason. ACL 2023. https://arxiv.org/abs/2212.08410
Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-J. and Pfister, T. (2023). Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Models. Findings of the ACL 2023. https://arxiv.org/abs/2305.02301
DeepSeek-AI et al. (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645:633–638. https://doi.org/10.1038/s41586-025-09422-z
Ye, H., Huang, X., Xiao, S., Chern, E., Xia, C. and Liu, X. (2025). LIMO: Less is More for Reasoning. arXiv:2502.03387. https://arxiv.org/abs/2502.03387
Muennighoff, N., Yang, Z., Shi, W., Li, J., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E. and Hashimoto, T. (2025). s1: Simple test-time scaling. arXiv:2501.19393. https://arxiv.org/abs/2501.19393