Architecting a Distillation Engine
What is the smallest complete, reproducible, and rights-first system capable of running a distillation workflow? A state-machine workbench wiring existing components, with an E1 replay trace and refusal paths.
Thirty-nine chapters built the parts: specifications, datasets, losses, metrics, teachers, students, evaluation, rights gates, safety checks, lineage, and audits. This chapter assembles them into the smallest system that can run a distillation workflow without lying about what it did:
What is the smallest architecture that can run a lawful, reproducible distillation experiment end to end?
The answer is a state machine with eleven transitions and no platform: a workbench that wires existing modules, refuses to train without permission, replays frozen evidence without relabelling it, and holds — rather than promotes — when a human has not authorised the outcome. The workbench is a reference learning system, not a product platform, and the chapter defends that boundary explicitly.
This chapter introduces no new concept. It integrates KD-032 (independent correctness, read not recomputed), KD-036 in preview (the promotion decision, fully defined in Chapter 42), KD-041 through KD-043 (provenance, lineage, rights as executable gates rather than prose aspirations).
What a distillation engine actually does
Strip away the scaffolding and a distillation engine performs five acts. It declares what will happen: teacher, student, data, objective, budget, and the action being authorised, all named before anything executes. It permits: a rights record covering the exact action, verified current, or nothing happens. It executes deterministically: seeded, single-threaded, CPU-first, with every input hashed and every output recorded. It measures independently of the teacher: held-out correctness reported separately from agreement, calibration alongside accuracy, cost alongside quality. And it decides on the record: promote, hold, or reject, with reasons that cite evidence rather than adjectives.
Three things it is not. It is not an experiment runner in the sense of a job scheduler: there is no queue, no retry policy, no distributed fleet — one run, one manifest, one lineage graph. It is not a research workbench in the exploratory sense either: nothing here tunes, searches, or iterates; the workbench executes a declared plan and otherwise refuses.
And it is emphatically not a production AI platform: no serving, no autoscaling, no user management, no deployment pipeline. The roadmap’s forbidden list — database, API server, queue, model registry, agent framework — remains forbidden, and the chapter records why each omission is load-bearing: every omitted component is a category of hidden state the audit would otherwise have to chase.
The components and their contracts
The workbench composes thirteen existing pieces, each with an
explicit input/output contract. Teacher and student come from the
specification layer (TeacherSpec, StudentSpec, compatibility
asserted, not assumed). The task contract fixes inputs, targets,
metric, and splits, so correctness has a denominator before
training starts. The rights audit answers one question per action
— permitted, prohibited, requires permission, or unknown — and
unknown blocks. The provenance validator checks dataset identity,
fingerprints, and contamination status against the manifest. The
dataset adapter materialises deterministic splits from recorded
seeds. The training adapter is, in this batch, a replay adapter:
it reads frozen checkpoints and logs rather than optimising, and
says so in every record it writes.
The evaluation adapter reads aggregate metrics without recomputing them, preserving the per-seed spread the book reports everywhere. Resource budgeting is a first-class record, not a footnote: the manifest carries the training budget (updates, batch size, seeds), the evaluation carries wall-clock training seconds where measured, and the benchmark record carries parameters, bytes, latency, and memory. The workbench refuses budget comparisons it cannot ground — two runs with different update counts are reported with both counts visible, never silently ranked — which is the Chapter 34 comparability rule compiled into the same control flow as the rights gate. Cost is a measurement with a method attached, and the method travels with the number or the number does not travel.
Calibration travels the same path: ECE alongside accuracy from Chapter 5 onward, so the gate never sees a correctness figure without its confidence audit. The Chapter 31 ledger rides along on every evaluation, so a student that is smaller on paper but identical in measured cost arrives at the promotion gate with both numbers visible. The safety assessment applies the Chapter 35 gate semantics. The experiment manifest is the durable artefact: run identity, config hash, source hashes, rights reference, results, limitations. The lineage graph links all of it, and the promotion assessment renders the seven-key summary Chapters 39 and 42 share.
The dependency rule is one-directional: the workbench imports the modules; no module imports the workbench. A component that cannot be tested without the pipeline is a component that has absorbed the pipeline, and the test suite enforces the separation — thirty focused tests in this batch, each naming the module under test, none requiring the others except through declared interfaces.
Interface contracts
Each component’s contract fits in one sentence plus its types.
configure(manifest_path, replay_id) takes a frozen manifest
path and a distinct replay identity and returns a CONFIGURED
run; it reads JSON and stamps nothing else. rights_check(run, records) takes the run and the rights-record list and returns
RIGHTS_CHECKED or REJECTED with the gate’s verbatim reason —
the refusal text is the interface, because downstream tooling
matches on it. validate_evidence(run, tamper=None) takes the
checked run and an optional tamper fixture and returns RUNNING
or HELD with every problem named; the tamper parameter exists
solely so failure paths are tested without touching protected
artefacts.
evaluate(run) returns EVALUATED with per-arm figures
read verbatim and recomputed: False stamped alongside them.
build_lineage(run) returns the same state with a ten-node graph
attached and its gap list logged. audit(run) returns the
seven-key promotion summary, field names identical to the Chapter
39 contract — any drift is a defect, and a test pins the keys.
decide(run, summary, safety, human_approval) returns one of the
three terminal states with reasons and evidence attached.
resume(manifest_path, replay_id) re-enters idempotently;
run_manifest(run) renders the durable record. No function
trains, downloads, serves, or deploys; the type signatures make
that absence checkable rather than asserted.
manifest.json + rights.json + replay_id
|
configure --> WorkbenchRun(CONFIGURED)
|
rights_check(records) --> RIGHTS_CHECKED | REJECTED
|
validate_evidence --> RUNNING | HELD (names every problem)
|
evaluate --> EVALUATED (reads aggregates, recomputed=False)
|
build_lineage --> 10-node graph + gap list
|
audit --> seven-key promotion_summary
|
decide(safety, human_approval) --> PROMOTED | HELD | REJECTED
flowchart LR
CFG[configure: manifest + replay id] --> RC[rights check: permitted or refused]
RC --> VE[validate evidence: hashes + arms]
VE --> EV[evaluate: read aggregates]
EV --> LG[lineage: 10-node graph]
LG --> AU[audit: seven-key summary]
AU --> DC{decide: safety + human}
DC -->|clear + approved| PR[PROMOTED]
DC -->|unresolved| HO[HELD]
DC -->|blocked| RJ[REJECTED]
Figure 1. The workbench pipeline as a gate sequence: every arrow advances or refuses, and there is exactly one path to RUNNING — through the rights check.
Rights-first, concretely
The enforced sequence — propose, identify assets, establish rights, validate evidence and resources, execute, evaluate, audit, decide — is the pipeline order, not a checklist taped beside it. Rights-first has three teeth. First, granularity: the gate authorises the manifest’s recorded action and nothing else, so a record permitting training never authorises redistribution or commercial deployment — the Chapter 36 twelve-action lesson compiled into control flow. Second, currency: review expiry is checked against the decision date, so a lapsed review refuses exactly as loudly as a missing record.
Third, unavoidability: there is exactly one path to RUNNING and it passes through the gate; the constructors offer no alternate route, and the order guards raise on any attempt to skip ahead. The E1 replay passes through this sequence with RM-101; the empty-records fixture demonstrates the opposite exit with the reason preserved verbatim. A gate that can be bypassed is decoration; this one is load-bearing because the code has no way around it.
The state machine
CONFIGURED --rights_check--> RIGHTS_CHECKED --validate--> RUNNING
--evaluate--> EVALUATED --decide--> PROMOTED | HELD | REJECTED
Seven states, three of them terminal (PROMOTED, HELD,
REJECTED); every arrow is a function
that either advances or refuses, never one that silently skips.
CONFIGURED means a frozen manifest is loaded and a distinct
replay identity minted — the replay is a new record about an old
run, and the identity says so. RIGHTS_CHECKED means exactly one
thing: check_action returned allowed for the manifest’s
recorded action against current records.
Anything else —
unknown record, expired review, mismatched action — lands in
REJECTED with the gate’s reason verbatim, before any data is
touched. RUNNING means the evidence connects: config hash
present, source hashes present (either key the manifest schema
uses), aggregate arms present. EVALUATED means the frozen
figures are read into an evaluation record stamped
recomputed: False. The terminal trio needs no elaboration here;
Chapter 42 defines their semantics, and the workbench implements
them without improvisation.
Failure handling is resumability, not recovery: any state can be rebuilt from the manifest plus the replay identity, so a crash loses no information that is not already versioned. There is no retry logic because there is nothing to retry — replay steps are deterministic reads, and a failed read fails identically on repetition, which is precisely what makes the failure diagnosable. Environment capture follows the same minimalism: the manifest records the platform, thread count, and dependency versions that produced the evidence, and the replay checks presence of those records without pretending the current machine is the original one.
A workbench that demanded identical hardware would never run anywhere twice; one that ignores the environment entirely cannot explain timing differences. Recording the environment and scoping timing claims to it is the middle path, and the E1 manifests’ host-sensitive latency notes are the precedent.
Worked E1 integration trace
The contract requires an end-to-end E1 run and a rights gate that
blocks a bad source. This batch completes the integration without
a new bounded training run: the replay executes every interface —
rights, evidence, evaluation, lineage, audit, decision — against
the frozen E1-classical manifest, and the refusal paths carry the
tested training-adjacent behaviour. The full trace is in
planning/40-evidence.md; the shape is:
| Step | State | Record |
|---|---|---|
| configure | CONFIGURED | manifest loaded; replay WB-E1-REPLAY-20261116 minted |
| rights_check | RIGHTS_CHECKED | RM-101 permitted for train_and_evaluate_e1 |
| validate_evidence | RUNNING | 16 code hashes, 8 arms, config hash present |
| evaluate | EVALUATED | 8 arms read; nothing retrained, nothing recomputed |
| build_lineage | EVALUATED | 10 nodes; gaps: none |
| audit | EVALUATED | promotion=CONDITIONAL |
| decide | HELD | human approval absent; automated checks cannot authorise themselves |
Read the terminal state carefully: the pipeline ran perfectly and the student was not promoted. The E1 replay ends HELD for exactly one reason — no qualified human authorised the outcome — which is the programme’s actual state (publication not approved, independent review outstanding) rendered as a machine verdict rather than a status page. Note the chain that produced it: the audit summary read CONDITIONAL (nothing failed, nothing unknown in the rights column, gaps documented), the safety input was PROMOTE for the replay’s synthetic purposes, and still the decision function returned HOLD — because these are conjunctive requirements, not votes, and one absent conjunct decides.
That is the workbench’s most important behaviour, and it is a refusal: everything the machine can check passed, and the machine declined to stand in for the human. A workbench that promotes on a clean technical run alone would be precisely the self-authorising system Chapter 42 forbids.
Teaching-signal selection is a configuration choice with a
provenance tail: the manifest records which signal supervised the
student (response KL, logit MSE, feature match, relational
preservation, behaviour cloning, preference objective), and the
lineage graph’s GENERATED_BY edges say where the teaching
signal itself came from — teacher logits, verifier filter,
human-ranked pairs. Two runs that differ only in signal are two
different experiments, not two seeds of one, and the workbench
treats them as such: the signal is part of the config hash, so a
signal substitution without a config change is a mismatch the
evidence validator holds.
This is where the book’s negative results live as engineering facts rather than prose warnings — the DPO objective that optimised preference while exact match stayed at zero (its arms never learned to generate) is, in workbench terms, a signal whose evaluation record refuses promotion, and the refusal is attached to the signal choice for any future run to read.
The reproducibility manifest for the replay is the run record itself: replay identity, manifest path, historic run reference, gate reason, event log, evaluation arms, lineage nodes and gaps, decision and reasons. Re-running the replay reproduces the record; resuming from the manifest reaches the identical terminal state, pinned in tests. What the replay does not do is train: there is no optimiser, no gradient, no checkpoint written, and the chapter refuses the chapter-40-shaped lie of presenting an integration exercise as an experimental result.
Failure trace
The contract requires a blocked run with an unknown rights status, a resumed run, and a failure injection that leaves lineage intact. All three are tested, and the second two are worth narrating because they show what “leaves lineage intact” means operationally.
An empty rights set refuses before running: state REJECTED,
reason record RM-101 not found, and validate_evidence raises
rather than proceeding — the order guard is a ValueError, not a
comment, so out-of-order invocation fails loudly in any future
caller. A tampered source hash holds at evidence validation with
the path named. A safety REJECT with human approval present
rejects while the lineage graph’s nodes and edges remain
byte-identical before and after the decision: the refusal is
recorded as a review outcome, not as an edit to history. Resume
from the manifest replays to the identical HELD state, showing
that the run record holds everything the replay needs to reach the
same decision.
The failure modes the architecture guards against map directly onto the contract’s list. Hidden state outside manifests: every step reads the manifest or the declared tamper fixture, and the tamper fixture is itself a declared input. A bypassable rights gate: the gate is a function call in the only path to RUNNING, with no alternate constructor. Configuration drift: config hash checked, mismatch held. Duplicated implementations: the pipeline imports its library modules directly (audit, promotion, lineage, rights) rather than reimplementing their logic, and its own job is sequencing.
Reproducibility from records
“Reproducible from its config, source commit, seeds, and manifests” is a precise claim with four checkable parts. The config hash pins every hyperparameter, budget, and split size; the source commit plus per-file hashes pin the code that interpreted them; the recorded seeds pin the randomness; the manifest binds the three together with the results they produced. The E1 replay verifies presence of all four and recomputes none of them — re-execution is prohibited by the programme’s evidence policy, so the workbench checks that the chain is present (hashes recorded, lineage complete) without claiming the experiment would reproduce bit-identically on new hardware.
That distinction matters: bit-reproducibility across machines was never established for these studies (host-sensitive timings are recorded as such throughout), and the workbench refuses to upgrade “complete records” into “guaranteed reproduction.” What re-running the replay does prove is narrower and more useful: given the same manifest, the pipeline reaches the same state, applies the same gates, and renders the same record — the resume test pins exactly this, and it is the only reproducibility claim the chapter makes.
Failed and incomplete workflows
The contract lists nine failure paths; the workbench
demonstrates eight refusals and one honest absence. Missing
training rights: REJECTED before data is touched. Incomplete
provenance: HELD at evidence validation with the missing field
named (the E1 manifest’s code_hashes-vs-source_hashes
adaptation is itself an example — two schema generations, both
accepted, the key used recorded in the event log). Invalid
dataset identities: HELD, because a fingerprint that matches
nothing is evidence of nothing. Mismatched evidence hashes:
HELD with the path named. Missing independent evaluation: HELD,
never inferred from agreement. Unsafe regression findings: the
decision function takes the safety verdict as input and REJECTs
on REJECT regardless of metrics — demonstrated with the frozen
E1 record plus an injected REJECT, lineage untouched.
Insufficient model-selection evidence: HELD, citing the missing comparison rather than ranking on vibes. Missing human approval: HELD, the E1 replay’s actual terminal state. Unresolved redistribution rights: REJECTED at the decision stage when the release under review requires redistribution — demonstrated in tests, pending in the programme. The ninth path, a genuinely new bounded training run, is the honest absence: this batch completes the integration without one, the contract’s “if authorisation and resources allow” condition unmet by neither but unneeded, and the chapter marks it not-measured rather than simulating a training loop and calling it E1.
What was deliberately omitted
The chapter’s most important section is a list of things not built, each with the audit-driven reason. No database: manifests are files, grep is the query language, and a queryable store would create a second source of truth the lineage graph would then have to audit. No job queue or orchestration: one run at a time, in the foreground, with the event log as the schedule. No model registry: checkpoints are git-ignored artefacts referenced by hash, and a registry would convert references into inventory needing its own provenance.
No deployment mechanism: promotion ends at a decision record, and anything that acts on that record is a different system with a different authorisation. No dashboard: the markdown reports are the interface, readable by humans and machines without a running process. No automatic retraining scheduler: a timer that re-executes training is a governance decision wearing automation clothes, and every re-execution needs the fresh rights review, fresh evaluation, and fresh human approval that a scheduler exists to skip. Count the moving parts: thirteen imported modules, one pipeline file, four focused test files. Anything more would be platform; the book needs a reference.
Limitations
The workbench replays one experiment family end to end; other families reuse the same adapters but have not each been replayed through it. The training adapter is a replay adapter — a bounded live training run remains unauthorised-unneeded rather than executed, and the chapter claims no training capability. Safety assessment consumes gate verdicts; it does not generate new safety evidence. Human approval is genuinely absent, which is honest rather than staged, and equally means the PROMOTED path through this pipeline is untested by design — no test promotes, because promotion requires a human the test harness must not invent.
Schema drift is the known fragility: manifests evolved
across batches (code_hashes in E1, source_hashes later, new
aggregate keys per family), and the workbench adapts by accepting
either key while recording which one it used — adaptation with an
audit trail, not silent coercion. A manifest from a future schema
the adapters do not know fails validation loudly rather than
parsing partially, which is the correct failure for a system
whose job is refusing to guess. No independent review; this
chapter is educational, not a deployment manual.
The artifact policy — versioned summaries, ignored checkpoints — deserves its final defence here, because it looks like an implementation detail and functions as a rights control. Checkpoints contain the full trained weights: versioning them would distribute the student’s parameters through the repository history, where the audit’s redistribution rows have no covering record. Git-ignoring them is therefore not tidiness but compliance architecture: what the repository keeps (metrics, hashes, configs, lineage) is publishable under its own terms; what it refuses to keep (weights) is exactly what no record authorises it to share. The policy converts a legal constraint into a filesystem rule, which is the cheapest kind of enforcement — one that works even when nobody is watching.
Repeated training is not continuous improvement
The contract requires this distinction, and the architecture enforces it. Re-running the same training with the same data produces the same student, modulo noise — that is reproduction, not improvement. Improvement requires a feedback loop the workbench deliberately lacks: monitoring a deployed student, detecting drift or regression, re-evaluating against current ground truth, retraining under a fresh rights review, and promoting through the full gate again.
Each loop iteration is a new experiment with new evidence, new permissions, and new human approvals — which is why Chapter 42’s system is a governance loop around this engine rather than a bigger engine. A pipeline that retrains nightly and promotes automatically is not continuously improving; it is continuously changing, and the difference between those verbs is the entire subject of the next two chapters.
Preparing the synthesis
The engine runs; the question it exists to answer is scientific, not architectural: what did forty-odd experiments actually establish? Chapter 41 turns the same manifests the workbench replays into the book’s capstone evidence report — four flagships, one synthesis table, and an explicit account of what remains unmeasured.
References
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H. and Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM 64(12). arXiv:1803.09010. https://arxiv.org/abs/1803.09010
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D. and Gebru, T. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT 2019).* https://doi.org/10.1145/3287560.3287596