Building a Legally Auditable Distillation Pipeline
Can an independent reviewer trace a student from its sources to its promotion decision? A minimal lineage graph, twelve audit questions, ten deterministic fixtures, and the E1–E4 integration audit.
Chapters 36–38 decided what is permitted, what is personal, and what is required. This chapter builds the machine that traces any of it — because a permission nobody can trace is a rumour, and a measurement nobody can reproduce is a story:
Can an independent reviewer trace a student from its sources and permissions to its evaluation and promotion decision?
The answer is a lineage graph with eight relations, a completeness score that fails loudly, an integration audit over all twenty-five book experiments, and a promotion summary that Chapters 40 and 42 consume directly. The audit’s verdict on this programme is BLOCKED — and the chapter explains why a blocked audit is the system working, not the system failing.
This chapter introduces KD-042 Model lineage (registry: the versioned chain from sources and teacher through training, evaluation, and deployment, including an audit trail). It is educational, not legal advice. A technical audit is evidence of process, never a legal guarantee.
What “auditable” means
An auditable pipeline satisfies four properties. Traceability: every artifact points to its antecedents — the dataset to its source and licence, the teaching output to its teacher, the student to its training run, the run to its authorising record. Completeness: required links are enumerated per node kind, and absence is reported as a gap, not smoothed over. Integrity: content hashes recorded at the time still match, so the reviewer knows the evidence has not changed (not that it was ever correct). Reviewability: an independent reviewer, given the graph and the manifests, can reconstruct each decision without asking the author what happened.
Reproducibility is the hard version of reviewability — re-running from the recorded commit, config, and seeds — and this book’s frozen manifests exist to make it possible, not to claim it has been done twice. The distinction matters because the two properties fail independently: a study can be reviewable but not reproducible (evidence complete, environment lost), or reproducible but not reviewable (re-runnable, with the decisions that made it so nowhere recorded). The book’s standing policy — reruns prohibited to protect frozen evidence — trades reproducibility checks for evidence stability, and the audit says so plainly: presence verified, recomputation deferred to the final review, whenever the programme decides the stability requirement has served its purpose.
The graph is deliberately minimal. Twelve node kinds (source
asset, licence, rights decision, dataset, teaching output,
teacher, student, training run, evaluation run, artifact,
deletion event, review decision) and eight relations
(DERIVED_FROM, GENERATED_BY, TRAINED_ON, EVALUATED_BY,
AUTHORISED_BY, SUPERSEDES, RETAINED_UNDER,
DELETION_DEPENDENCY). No database, no service, no registry
process: JSON-serialisable nodes and edges with version and
content-hash requirements on every node.
The relation semantics
are fixed in one place — edges point from the dependent to its
antecedent, except SUPERSEDES (old to new) and
DELETION_DEPENDENCY (derived to retirement event) — because a
graph whose arrows mean different things on different days is a
drawing, not an audit instrument. Model cards (Mitchell et al.,
P-044) and datasheets (Gebru et al., P-043) supply the
documentation practice behind the node contents: intended use,
provenance, and caveats recorded where the artifact lives.
The twelve questions
The system exists to answer twelve questions. Each maps to a traversal, and the chapter demonstrates every one:
- Which teacher produced this training data? — ancestors of the
dataset through
GENERATED_BY. - Which sources contributed to it? — ancestors through
DERIVED_FROMtoSOURCE_ASSETnodes. - What permissions covered each action? —
AUTHORISED_BYedges to rights decisions with their recorded scopes. - Which student checkpoints used it? — dependents of the dataset
(
TRAINED_ONreversed). - Which evaluation results support the student? —
EVALUATED_BYedges from evaluation runs to artifacts, with metrics attached. - Which results are incomplete or missing? —
missing_linksagainst the per-kind required relations. - What happens if a source must be retired? —
retirement_impact: every dependent enumerated for re-evaluation or retirement. - Which datasets and models depend on that source? — same traversal, reported as the blast radius.
- Is there evidence that required deletion or retirement
completed? —
DELETION_EVENTnodes with trails; absent trail means absent evidence. - Is the proposed redistribution or deployment action authorised? — scope check on the authorising record; unknown scope blocks.
- Which decisions require a human reviewer? — every
REQUIRED/UNKNOWNstatus and every HOLD finding, listed. - Which claims are not supported by available evidence? — the completeness score’s gap list, verbatim.
Questions 6, 9, 10, and 12 are where audits usually die: the missing result nobody lists, the deletion nobody verifies, the redistribution nobody authorised, the claim nothing supports. Question 12 deserves a final emphasis because this book practices it in prose: the retrieval gain reported with its per-seed spread rather than rounded up, the 0.306³ compounding shown as an illustration beside the measured zero rather than as an explanation of it, the pruning null kept visible instead of tuned away.
An audit that cannot represent “measured, bounded, and insufficient” will round every one of those into a success given time — which is why unsupported-claim detection is a query, not a vibe: list every reported number, join it to its denominator and provenance, and print what is left over. The leftovers are the audit’s most valuable output. The graph answers them by construction — a gap is a first-class output — which is why the completeness score is a fraction with names attached rather than a badge.
Ten fixtures, ten lessons
Fixtures A–J (examples/common/audit.py) are synthetic and
self-contained; each builds a small graph and asserts the finding.
Expected verdicts are pinned in tests, so the lessons cannot drift:
A — complete permitted lineage: PASS. Eleven nodes, twelve edges, completeness 1.00, all hashes verified. The reference shape every real audit is measured against — and the disposition is careful: promotable within stated synthetic coverage, never certified.
B — missing source provenance: FAIL. A dataset with no
SOURCE_ASSET ancestor. Disposition: reconstruct provenance or
quarantine the dataset. This is the E3-fingerprint gap in
miniature (below).
C — missing rights record: FAIL. Training and dataset nodes
without AUTHORISED_BY. Disposition: no training without an
authorising record — the gate Chapter 36 built, now as a graph
property.
D — inference-only permission: FAIL. The authorising scope says inference; the action is training. Disposition: permission does not radiate. The most common real-world shape of B-through-D is one record read too broadly.
E — revoked permission with dependents: HOLD. A superseding licence leaves three nodes still referencing the old terms. Disposition: re-evaluate every dependent under the new terms or retire them. Revocation is a graph event with a blast radius, not a memo. Programme relevance: every provider-terms update and every model-card relicensing is a potential Fixture E, which is why the audit records licence versions rather than names — “Apache-2.0” is not a version, and a relicensed dependency replays this fixture at full scale.
F — deleted source, live derived dataset: FAIL. The deletion event exists; the derived dataset was never withdrawn. Disposition: deletion must reach derived artifacts — the Chapter 38 ladder as a graph check.
G — checkpoint without training lineage: FAIL. A student with
no DERIVED_FROM. Disposition: a checkpoint without lineage is
not auditable, however good its metrics.
H — intact frozen hashes: PASS. Eleven hashes verified, zero mismatches. Disposition states the boundary: hashes prove the chain is intact, not that results are correct.
I — redistribution without permission: FAIL. Scope covers training only. Disposition: unknown is not permission — the Chapter 36 rows 8–9 as a fixture. Programme relevance: this is the fixture the book itself would fail if publication were proposed today, with the publisher sign-off outstanding — the audit’s most honest mirror, and the reason the batch report lists it as a blocker rather than burying it in limitations prose.
J — excellent metrics, failed safety gate: FAIL. Accuracy 0.92 with a critical-slice rejection. Disposition: metrics never overrule a failed safety check — the Chapter 35 gate, wired into the audit output Chapters 40 and 42 consume.
The E1 complete lineage (worked example)
The contract requires a complete lineage graph for E1. Assembled
from the frozen E1-classical manifest: self-created spiral source
(RM-101, VERIFIED_PERMITTED, IE/EU) → dataset with recorded
sizes, noise, and split seeds → teacher runs (17,412 parameters)
and student runs (228 parameters, eight arms, three seeds) →
per-seed metrics with agreement kept separate from correctness →
aggregate report with the negative accuracy result preserved.
Every element the graph requires is present in the manifest:
run identity, rights record with action, config and source hashes,
dataset description, per-seed results, limitations.
The arms
themselves become student nodes with distinct training runs:
A-oracle on ground-truth rows, B-teacher on unfiltered teacher
targets, C-filtered on verifier-selected rows — each with its own
TRAINED_ON edge to the filtered or unfiltered dataset variant,
each sharing the AUTHORISED_BY edge to RM-117, each evaluated by
the same held-out split. That fan-out is why lineage matters even
when results look simple: the headline “0.402 beats 0.367” is three
different training histories, and only the graph says which rows
each history saw. E1 is the book’s best-documented study precisely
because it was the first —
and the audit’s completeness check confirms it mechanically
rather than admiringly.
flowchart LR
SRC[source + licence] --> DS[dataset + manifest]
DS --> TR[teacher training run]
DS --> ST[student training run]
TR --> ST
RR[rights record] -.->|AUTHORISED_BY| TR
RR -.->|AUTHORISED_BY| ST
ST --> EV[held-out evaluation]
EV --> RP[report + promotion decision]
DEL[deletion event] -.->|retires| DS
Figure 1. The E1 chain in miniature: every arrow is a checkable edge, every node carries identity, version, and rights status. Simplified view; the implementation records twelve node kinds.
The E1–E4 integration audit
The full report is planning/39-e1-e4-lineage-audit.md,
generated read-only from all twenty-five experiment manifests. The
headline numbers: twenty-five experiments audited; every rights
status VERIFIED_PERMITTED for its recorded action; source-hash
counts and config-hash presence recorded per manifest; dataset
fingerprints present where the study design recorded them.
Evidence integrity is stated exactly: hashes recorded per manifest,
never re-verified, because reruns are prohibited — the audit
checks presence and consistency, not recomputation.
Per-family notes, read from the manifests: E1-classical carries dataset descriptors with split sizes, noise, and seed offsets plus teacher/student run records; E2-retrieval pins its corpus fingerprint with per-arm latency and ranking metrics; E3 studies record source fingerprints per seed where the design included them (structured, transformer, preference) and omit them where it did not (reasoning, verifier, on-policy, synthetic-generation, filter-curriculum, peft); E4-agent records trajectory fingerprints per seed under RM-118; E4-memory records the store fingerprint with its deletion audit (live 60, retired 4) under RM-119; E31-benchmark records fourteen inference architectures with no training lineage by design. Aggregate-style manifests (E1-blackbox, E2-ensemble, E1-classical’s group record) carry no per-seed arrays; the audit counts their seeds as zero and their hashes as present rather than imputing a seed structure they were never designed to have — absence of a field the design never promised is not a missing link. Each note is a presence claim about the manifest, checkable with the audit script, and nothing more.
Two documented gaps carry forward, neither re-opened. First, the
D-FP-11 missing-run-history limitation: the audit surfaces it
from its planning record and stops there, as instructed — visible,
not repaired, not re-investigated. Second, seeds without
recorded source fingerprints in fifteen experiment families. The
audit counts only the source_fingerprint field, and several of
these families record other fingerprints instead — E2-retrieval its
corpus and splits, E4-agent its trajectories, E4-memory its store.
They are reported as missing-link findings (the Fixture B pattern at programme scale), explicitly not defects to repair, because repairing them would mean re-running frozen studies. A pipeline with excellent metrics but missing training rights would stay blocked for the affected action; here the rights are complete and the missing items are fingerprints, so the finding is incompleteness, not prohibition — the audit distinguishes the two because the consequences differ.
Review dependencies are outstanding by policy: publication not approved, independent review pending. Deletion dependencies rest on the unit-scoped MemoryStore demonstration; no lineage-linked erasure pipeline exists yet. The promotion summary therefore reads BLOCKED — with the blocking set named (B, C, D, E, F, G, I, J are fixture demonstrations; the programme-level blocks are review approval, redistribution permission, and erasure machinery). A blocked summary with named blockers is the deliverable, not a failure of it: Chapters 40 and 42 consume exactly this shape, and a summary that cannot say BLOCKED cannot be trusted to say PROMOTE.
The promotion interface
promotion_summary returns seven keys for Chapters 40 and 42:
scientific sufficiency, rights position, privacy-review position,
safety-gate reference, reproducibility evidence, artifact
integrity, and human-approval status — plus the overall decision,
the blocking set, and the carried gaps. In the frozen programme
audit the scientific key reads sufficient-with-limits, rights
reads permitted-for-recorded-actions-only, privacy reads
required-for-personal-data but not-applicable to the synthetic
inventory, safety defers to the Chapter 35 REJECT/HOLD semantics
for the illustrative fixtures, reproducibility cites frozen
manifests with recorded hashes, integrity cites hash presence
without re-verification, and human approval reads outstanding —
yielding BLOCKED overall with the blocking set named.
Unknown evidence blocks the affected action by construction, in every case. The interface is machine-readable markdown and JSON-compatible dicts, versioned with the graph schema, and deliberately boring: Chapters 40 and 42 need a contract, not a ceremony. The legal chapter’s contribution ends where the engineering chapters’ begins — at a function signature both sides can test, version, and trust.
Failure modes of the audit itself
An audit system fails in six familiar ways, each guarded here.
Orphan datasets and checkpoints — nodes with no antecedents —
are Fixture B findings, not mysteries. Missing licence
versions are node-validation errors: version is required, and
“latest” is not a version. Silent overwrites are prevented by
append-only discipline: reruns mint new run IDs (the book’s
standing rule), corrections are new records, and SUPERSEDES
links the versions. The book practices this visibly: the
E3-reasoning study carries five protocol amendments as separate
files alongside the original protocol, each expanding the design
without editing it — the graph equivalent is a new protocol node
with a SUPERSEDES edge to its predecessor, so a reviewer sees
the whole design history instead of the final design pretending
to be the only one.
An audit that cannot show its own revisions cannot be trusted to show anyone else’s. Deletion that misses derived artifacts is Fixture F. A graph recording names without versions fails node validation at entry. A log file mistaken for an audit is the deepest failure: logs record events, audits answer the twelve questions — and any report that cannot answer question 12 (which claims lack support) is a log with ambitions. The chapter’s remedy is uniform: represent the absence, name the consequence, block the action.
The audit record itself: retention and access
The audit record may contain personal or confidential information — subject identifiers in deletion logs, commercial terms in licence nodes, reviewer names on every decision — so retention and access controls apply to the audit itself (L-021). The rule is symmetric: the audit keeps, for each artifact, what was decided, by whom, on what evidence, for as long as review and legal duties require; and it restricts that record to reviewers and counsel with the same authentication discipline Chapter 37 records for research copies.
Deletion requests reach the audit trail too — the trail records that an erasure occurred and what it covered, without retaining the erased content to prove it. An audit system with world-readable logs is a disclosure incident wearing a compliance badge.
A reviewer reconstructs a promotion decision (worked example)
The contract requires a reviewer reconstructing a promotion decision. Take the E3-structured student, run E3S-20261116T115159Z-1479dc, arm C-filtered at 0.402 functional correctness. The reviewer starts from the promotion summary (BLOCKED, with the blocking set named) and walks backwards. Scientific: sufficient-with-limits — three seeds, matched budgets, the filtering delta exceeding its spread. Rights: permitted-for-recorded-actions-only — RM-117 covers training, nothing covers redistribution, so the release under review must be an internal deployment or stay blocked. Privacy: synthetic inventory, no review required. Safety: the Chapter 35 gate for this arm — format-valid, functionally measured, no slice failure recorded.
Reproducibility: frozen manifest with config and source hashes; reruns prohibited for verification, so the reviewer checks presence, not recomputation. Artifact integrity: hashes recorded. Human approval: outstanding — and there the reconstruction stops, correctly: the reviewer can verify every technical claim and must still refuse promotion pending the author, independent, and publisher approvals the summary names. The worked example shows the audit doing its real job — making the one missing thing unmistakable — rather than the imagined job of replacing judgement.
A deletion propagation record (worked example)
The contract requires a deletion propagation record. Scenario, on synthetic data: the E4 memory store retires four KB combinations (live 60, retired 4, per the E4M manifest’s deletion audit). The record lists: the deletion event with its trail (which keys, which seed, which purpose); the read-through verification (retired keys return None on every subsequent lookup); the dependents enumerated from the graph (students trained on the pre-deletion KB, evaluation reports citing them); and the disposition per dependent (re-train, re-evaluate, or annotate as superseded).
What the record does not contain is a parametric-erasure claim — the student weights are unchanged by the store deletion, the ladder rung stays NOT VERIFIED, and any reviewer reading “deletion complete” into this record is misreading it. Propagation records end at the boundary of what was demonstrated; the boundary is part of the record.
Completeness, worked arithmetically
The contract asks for the completeness score defined and a
missing-link example shown. Take a training run with a dataset but
no authorising record: required outbound relations for
TRAIN_RUN are TRAINED_ON and AUTHORISED_BY; one of two is
present, and missing_links reports
run (TRAIN_RUN): missing AUTHORISED_BY. At graph scale the score
is the fraction of nodes with all required links present — the
Fixture A reference graph scores 1.00 across eleven nodes, while
any real audit scores below it by exactly the named gaps. The
number is not a grade; it is an index into the gap list, and the
gap list is the work plan.
AI Act record-keeping, placed
training-content-summary duties (L-023, with the Article 113 caveat carried from Chapters 36–38) describe records this graph already keeps in miniature: what was built, from what, under which permission, with what measured result. The chapter claims no more than structural readiness: the node kinds cover the dossier shape, the rights links cover the copyright-policy pointer, and the hash discipline covers integrity. Whether this programme is an obligated provider is the Chapter 38 role analysis, unchanged — the graph serves any verdict without prejudging it.
Limitations
The graph models evidence, not truth: complete lineage of a flawed study is still a flawed study. Fingerprint gaps in fifteen families bound what the integration audit can claim — the audit lists exactly which families (from D30-diffusion through E4-memory, visible in the report table’s Missing FP column), so a reviewer counting on per-seed fingerprints knows where not to lean (bearing in mind that some of them carry corpus, trajectory, or store fingerprints the audit does not count). Deletion machinery is unit-scoped.
The D-FP-11 gap is carried, not closed. Deterministic fixtures demonstrate the contract at eleven nodes, not production scale: a thousand-node programme graph should need the same relations and stricter indexing rather than new semantics, though that is untested. No independent review; this chapter is not legal advice. The audit’s own completeness, scored on the integration graph, is deliberately left for the reviewer to compute — the tool reports the gaps, the human judges whether the programme may proceed with them.
Preparing the engine
Part VII ends with the ledger complete: permission recorded (36), data rights recorded (37), privacy mapped (38), lineage traceable (39) — each with its blocked rows owned and waiting, each with its evidence frozen and its gaps named. Chapter 40 inherits all of it as an interface: the distillation engine that trains, evaluates, gates, and audits in one loop, with every refusal in this part available as a function call.
References
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H. and Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM 64(12). arXiv:1803.09010. https://arxiv.org/abs/1803.09010
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D. and Gebru, T. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT 2019).* https://doi.org/10.1145/3287560.3287596
Legal sources
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). In force 1 August 2024; general-purpose AI obligations from 2 August 2025. https://eur-lex.europa.eu/eli/reg/2024/1689/oj (retrieved 12 November 2026). Cited as L-023 for the documentation and record-keeping obligations relevant to the audit graph; dates subject to the Article 113 caveat; not legal advice.
Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data (General Data Protection Regulation). Applicable 25 May 2018. https://eur-lex.europa.eu/eli/reg/2016/679/oj (retrieved 12 November 2026). Cited as L-021 for the retention, security, and records duties behind audit retention and access controls; not legal advice.