Distillation From First Principles
Teaching AI to Learn from AI. A first-principles engineering guide to knowledge distillation: knowledge transfer, compression, specialisation, reasoning, evaluation, and lawful improvement of AI systems.
Distillation From First Principles is an engineering book about transferring capability from larger AI systems to smaller, cheaper, or more specialised ones. It develops the mathematics, implements the methods, measures what the student actually learns, and treats the legal right to use a teacher or a dataset as part of the engineering problem.
The book is in development. The chapter architecture is being established; drafts will appear here only after their sources, code, and evidence have been checked.
Chapters
What Is Knowledge Distillation?
Knowledge distillation uses a teacher model's behaviour as a learning signal for a student. This chapter defines the idea, separates transfer from imitation, and introduces the first controlled experiment.
Distillation Versus Other AI Improvements
Distillation is a learning mechanism, not a synonym for compression or improvement. This chapter separates the four transfer families from fine-tuning, self-training, reinforcement learning, quantisation, pruning, merging, and retrieval.
The Teacher and the Student
A distillation pair is viable only when the teacher's signal, the student's capacity, and the interface between them are compatible. This chapter defines the roles, the access regime, capacity, and the E1 pair.
Probabilities, Logits, and Soft Targets
A teacher's final layer returns scores, not certainties. This chapter derives softmax, temperature scaling, cross-entropy, and the logit gradient, then connects them to the tested E1 implementation.
Deriving the Distillation Objective
The teacher's distribution becomes a training target through KL divergence and a temperature-scaled loss. This chapter derives the objective, checks it against E1, and separates agreement from correctness.
Response-Based and Logit Distillation
Teacher outputs can be matched as probabilities, logits, rankings, or decoupled target and non-target terms. This chapter compares those signals, derives their invariances, and reads the E1 arm matrix honestly.
Black-Box and API-Only Distillation
Restricted teacher access changes the signal, objective, query cost, and legal obligations. This chapter formalises top-1, top-k, sampled, and full-probability access, derives truncation and sampling bias, and reports a bounded local E1 extension.
Feature-Based Distillation
Intermediate representations can carry structure that final outputs discard, but feature matching requires a projection, a layer correspondence, and a separate test of task correctness. This chapter derives feature MSE and cosine losses and reports the bounded E2 experiment.
Attention and Relational Distillation
Structure can transfer without copying values: spatial attention maps, pairwise distances, triplet angles, and similarity matrices. This chapter derives those objectives, shows their invariances and batch dependence, and reports a controlled E2 extension.
Sequence-Level Distillation
When the supervision target is a whole generated sequence rather than a token distribution, the objective, the decoding algorithm, and the failure modes all change. This chapter derives sequence likelihood, beam search, and length and exposure bias, and reports a bounded E3 precursor.
Ensembles and Multiple Teachers
When several teachers supervise one student, the aggregation rule is the method: probability averaging, logit averaging, and the KL mixture identity behave differently, and the best ensemble is not automatically the best teacher. This chapter derives the identity, measures disagreement, and reports a bounded two-teacher E2 extension.
The Data Behind the Knowledge
Distillation is only as sound as its data: origin, identity, splits, contamination, coverage, and permission. This chapter defines dataset provenance, builds a manifest, and retrospectively audits the E1 and E3 precursor datasets without rerunning them.
Constructing Teaching Tasks
A task definition decides what the student should learn: inputs, outputs, correctness, metrics, leakage controls, and difficulty. This chapter builds TaskSpecs for E1 and the E3 precursor, plus a small instruction task, with prompt-leakage and ambiguity checks.
Synthetic Data Generation
Generating teaching examples at scale requires measuring what generation produces: correctness, duplicates, diversity, coverage, cost, and provenance. This chapter builds a bounded E3 generator with an audited quality comparison and no student training.
Data Filtering and Curriculum Learning
Filtering decides which examples remain; curriculum decides the order they are presented. This chapter separates the two interventions, derives their effect on the training distribution, and tests them in a controlled E3 ablation with real student training.
Dataset Distillation
Dataset distillation optimises the training examples themselves so a small synthetic set can train a fresh student. This chapter derives the bilevel objective and gradient matching, compares trajectory and distribution matching, and runs a bounded E2-shape condensation demo.
Distilling Transformers and Instruction Models
Moving distillation from classifiers to decoder-only transformers: tokenisation, causal masking, completion-only loss, token-level versus sequence-level supervision, and a controlled E3 baseline with exactly matched update budgets.
Distilling Reasoning
Do worked procedures teach reasoning, or only its format? This chapter builds a checkable arithmetic trace task, compares answer-only against trace supervision with matched update budgets, and measures what transfers to longer problems.
Verifier-Guided Distillation
A verifier scores candidate reasoning traces and selects training data: outcome versus process checks, false positives, rejection sampling, best-of-n, and a controlled E3 comparison of unverified, outcome-verified, and process-verified supervision.
On-Policy Distillation
Train the student on its own generated trajectories with teacher supervision: forward and reverse KL, the distribution-mismatch argument, detached-prefix gradient surrogates, and a controlled E3 comparison of offline, on-policy, and stale-replay distillation.
Self-Distillation
Can a model improve by learning from a version of itself? Born-again self-distillation with soft targets, pseudo-label self-training with confidence thresholds, confirmation bias measured as error inheritance, and a controlled E1 comparison including a hard-teacher control and an external-teacher reference.
Preference and Alignment Distillation
Can a student learn preferred behaviour without confusing teacher preferences with independently correct answers? Bradley–Terry models, the DPO objective with a frozen reference, and a controlled E3 comparison of SFT, clean DPO, and noisy DPO.
Continual and Adaptive Distillation
Can a student keep learning from new teachers without forgetting what it already knew? A bounded two-task E1 study comparing sequential fine-tuning, bounded replay, and distillation regularisation — with retention policy, deletion, rollback, and a failed joint-training reference.
Classification, Embeddings, and Retrieval
Can a compact bi-encoder preserve a cross-encoder teacher's ranking behaviour while retaining independently measured retrieval relevance? Contrastive objectives, score distillation, hard negatives, Recall@k, MRR, nDCG, and a bounded E2 study with a weak, seed-concentrated result.
Coding and Structured Generation
Can a student learn to generate functionally correct structured outputs from teacher demonstrations rather than merely copying syntax? Constrained sequence likelihood, schema validation, test-based verification, and a bounded E3 study where verification filtering recovers the teacher-demonstration gap.
Tool Use and Agent Trajectories
Can a student learn to complete multi-step tasks by distilling a teacher's tool-use trajectories? Agent policies, behaviour cloning, compounding errors, distribution shift, recovery probes, and a bounded E4 study where cloning memorises but never generalises.
Distilling Memory and Procedural Skills
Should an agent acquire procedural knowledge through distillation into its parameters, retrieval from external memory, or both? A bounded E4 extension where retrieval and distilled weights tie on static accuracy, stale knowledge defeats both, and deletion plus read-through wins.
Multi-Agent and Expert Distillation
Can two specialised teachers transfer complementary capabilities to one student? Scripted specialists, a learned router, oracle-routing diagnostics, and a bounded E4 study where routing succeeds everywhere and trajectory execution almost nowhere.
Vision, Speech, and Multimodal Distillation
What transfers between modalities, and when does preserving an embedding relationship fail to preserve capability? A bounded image-to-signal E2 extension where relational supervision shapes representations without moving accuracy.
Distillation for Generative Media
Can a generative student produce useful samples with fewer inference steps than its teacher? Progressive halving on a 2-D mixture, a naive-striding baseline, and a null result: on this toy, step reduction costs nothing and distillation adds nothing.
Choosing the Right Student
How do we choose a student when correctness, latency, memory, and cost compete? A one-host inference benchmark, per-family Pareto analyses, and a constrained-selection rule that picks the calibrated student over the accurate one.
Parameter-Efficient Distillation
Can a student acquire teacher behaviour while updating far fewer parameters? LoRA adapters on a frozen base, a two-point rank sweep, exact merging, and a clean negative: with a random frozen base, the adapter fails where full fine-tuning succeeds.
Distillation, Quantisation, and Compression
When distillation is combined with quantisation or pruning, what actually gets cheaper? A bounded E1 demo where INT8 quantisation shrinks stored weights 3.5-fold with no latency change, and 50% pruning changes nothing but the accuracy ledger.
Evaluating Knowledge Transfer
What evidence justifies the claim that a student acquired a capability from a teacher? A transfer taxonomy, the mathematics of uncertainty, and a synthesis of every E1–E4 finding through one evaluation lens.
Failure Modes and Safety Regressions
How can a distilled student look better on average while being worse where it matters? Six deterministic safety fixtures, a machine-readable regression report, and a promotion gate that rejects on a failed slice.
Model Licences, Contracts, and Permitted Use
What exactly does the recorded agreement permit? A tabletop audit of E1–E4 across twelve actions, a licence-comparison matrix from Apache 2.0 to provider terms, and five hypothetical cases that stay hypothetical.
Copyright and Training-Data Rights
When can copyrighted material become teaching data? The EU text-and-data-mining framework, Ireland's implementation, database and software rights, and a read-only audit of the E3 teaching-data lineage.
Privacy, GDPR, and AI Regulation
Which personal-data and regulatory obligations follow data through a distillation pipeline? Lawful bases, minimisation, erasure across stores and derived artefacts, the AI Act's consolidated timetable, and a deterministic E4 privacy audit.
Building a Legally Auditable Distillation Pipeline
Can an independent reviewer trace a student from its sources to its promotion decision? A minimal lineage graph, twelve audit questions, ten deterministic fixtures, and the E1–E4 integration audit.
Architecting a Distillation Engine
What is the smallest complete, reproducible, and rights-first system capable of running a distillation workflow? A state-machine workbench wiring existing components, with an E1 replay trace and refusal paths.
Four Controlled Distillation Experiments
What do the book's E1–E4 experiments jointly establish? Four flagships with baselines and uncertainty, a Vera-level synthesis table, ten classified cross-experiment findings, and an explicit account of what remains unmeasured.
Deploying a Continuously Improving AI
How could a distilled system improve without silently degrading? A promotion gate that refuses, monitoring with declared thresholds, seven worked scenarios, a rollback simulation, and the human governance that makes improvement lawful.