Deploying a Continuously Improving AI
How could a distilled system improve without silently degrading? A promotion gate that refuses, monitoring with declared thresholds, seven worked scenarios, a rollback simulation, and the human governance that makes improvement lawful.
Forty-one chapters established what the students learned, what they cost, what risks they carry, and whether their use is authorised. One question remains, and it is the only one that matters outside the laboratory:
How could a distilled model be deployed and improved continuously without silently degrading correctness, safety, provenance, rights, or human control?
The answer is a gate that can say no, monitoring that can change its mind, a rollback that restores the past, and humans who own every step. This chapter introduces KD-036 Promotion gate (registry: pre-defined criteria a student must meet before deployment, covering correctness, safety, rights, and reproducibility) — and demonstrates it entirely by simulation. No live deployment is authorised, none is performed, and nothing here promotes the book’s own students.
A candidate with unresolved required rights, a critical safety failure, invalid evidence, or missing mandatory approval must not be promoted. That sentence is the chapter in miniature. Everything below is its machinery.
The promotion interface
Chapter 39 defined the seven-key summary; this chapter consumes
it verbatim — scientific, rights, privacy_review,
safety_gate, reproducibility, artifact_integrity,
human_approval — plus the promotion field and the blocking
set. Field names are not reinvented; any drift is a defect, and a
test pins the schema. The decision function combines three
inputs: the audit summary, the safety gate verdict, and the human
approval identity. Its semantics are fixed:
- PROMOTE only when the summary is clear, the safety gate passes within its stated coverage, and a qualified human has authorised the specific release. All three, conjunctively.
- HOLD when nothing is blocked but something is unresolved — a WARN finding, an inconclusive column, a missing approval. The release waits; waiting is a decision with a record.
- REJECT on any blocking finding: safety FAIL, missing or prohibited rights, invalid evidence, or an audit-level BLOCKED that this module never upgrades. Rejection is final for the release under review, and the rejected student remains evidence.
Two rules govern the interface. First, an UNKNOWN value is never
converted into a PASS — unknown blocks the affected action, full
stop, exactly as Chapter 36’s gate requires. Second, the Chapter
39 BLOCKED verdict is preserved, never reinterpreted: the
workbench that produced it and the promotion function that
consumes it agree that a blocked audit is a finding, not a
request for a second opinion. The render_decision output states
the verdict, the release, the reasons, and the evidence — and
carries the same caveat every report in this book carries: a
PROMOTE attests that the recorded gates passed, not that the
system is approved, compliant, or safe.
flowchart TD
S[summary + safety verdict + human identity] --> U{any blocking finding?}
U -->|safety REJECT, bad rights, invalid evidence| R[REJECT: final for this release]
U -->|nothing blocked| W{anything unresolved?}
W -->|WARN, inconclusive, no approval| H[HOLD: the release waits]
W -->|clear + passing safety + qualified approval| P[PROMOTE: conjunctive, documented]
Figure 1. The promotion decision as a gate: blocking findings reject, unresolved findings hold, and only the conjunction of all three inputs promotes.
Seven scenarios
Each scenario is worked end to end through decide. Scenarios
marked synthetic use fully synthetic candidates; scenarios
marked frozen reuse the book’s measured evidence with labelled
synthetic events where the mission requires them. No scenario
claims an actual historical candidate passed gates it did not.
| Scenario | Tests | Decision |
|---|---|---|
| A — better candidate, all gates pass | conjunctive promotion | PROMOTE |
| B — better average, worse slice | safety overrule | REJECT |
| C — rights missing for the release | permission does not radiate | REJECT |
| D — drift after promotion | monitor, rollback, incident | ROLLBACK |
| E — erasure uncertainty | parametric rung NOT VERIFIED | REJECT |
| F — agent aux metrics without episodes | episode floor binds | REJECT |
| G — no human approval | waiting is a decision | HOLD |
Table 1. The seven scenarios indexed: each names the conjunct that decides it. Details below.
Scenario A — Better candidate, all gates pass (synthetic). A synthetic student beats its baseline on held-out correctness (0.71 vs 0.64 on n = 400, drop band ±0.047 — the gain clears noise) with complete rights, a passing safety gate with no WARN findings, intact evidence, and a recorded human approval naming the release. Decision: PROMOTE. The scenario exists to define what promotion looks like when it happens — conjunctive, documented, boring — so that every other scenario shows exactly which conjunct fails.
Scenario B — Better average, worse critical slice (synthetic). Mean correctness rises while the critical slice falls past its floor; the safety gate returns REJECT. Decision: REJECT, regardless of the average. This is Fixture A of Chapter 35 wired into the promotion function, and the triggering text is preserved verbatim: accuracy never overrules a failed safety check.
Scenario C — Rights missing (frozen + synthetic event). A technically strong model — take the E3-structured filtered student at 0.402 functional correctness — lacks redistribution permission for the proposed release. The rights column reads unknown-for-that-action; the decision is REJECT for redistribution while training-adjacent uses remain covered. Permission does not radiate, and the scenario demonstrates the most commercially important refusal in the book.
Scenario D — Drift after promotion (simulated stream). A promoted synthetic candidate’s monitoring windows show correctness sliding past the degradation threshold beyond the noise band. Decision path: ALERT on the window, investigation, ROLLBACK to the approved version, incident record, mandated post-mortem and re-review before any re-promotion. The simulation is worked in full below.
Scenario E — Privacy and erasure uncertainty (frozen). A candidate checkpoint with unresolved deletion obligations: a MemoryStore retirement cleared the live keys while the distilled checkpoint trained on the pre-retirement data remains in the candidate set with no lineage-linked erasure behind it. The privacy review reads required-but-unresolved; the ladder rung for parametric erasure reads NOT VERIFIED.
Decision: REJECT for any action the uncertainty touches — the checkpoint may not serve the affected slices, and no retrieval-miss test on the store changes that, because the store and the weights are different objects with different deletion stories. Deletion from an external memory store is never treated as proof of erasure from model weights — the Chapter 38 boundary, enforced as a gate outcome rather than repeated as advice.
Scenario F — E4 agent failure (frozen). A student with promising action agreement (0.306) or perfect routing accuracy (1.000) but zero complete-episode success. Decision: REJECT — not promoted on the auxiliary metric alone, with the episode floor named as the binding criterion. The book’s zeros become the gate’s clearest refusals.
Scenario G — Missing human approval (frozen programme state). All automated checks pass; no qualified authorisation exists — exactly the E1 workbench replay’s terminal state. Decision: HOLD. The release waits for a human who does not yet exist in the record, and no passing test, clean gate, or intact hash changes that. This is the scenario the book itself lives in, and the gate states it plainly rather than promoting around it.
Monitoring: thresholds before data
Monitoring begins before the first observation, with thresholds
declared in writing: minimum correctness per slice, incident
allowance (zero, in this book’s configuration), maximum drop
versus baseline, noise band handling, latency budget, and the
rights status required for continued serving (VERIFIED_PERMITTED
— anything else blocks serving, not just promotion). Each
observation carries candidate identity, model version, period,
task and slice identifiers, correctness with its denominator,
safety incidents, drift indicators, resource metrics, rights and
privacy status, and an evidence reference. A window that cannot
state its denominator is not monitored; it is merely watched.
Drift detection uses one statistic, honestly bounded: the observed drop against baseline with a binomial noise band at the baseline rate. Worked through: baseline correctness 0.64 on n = 400 gives a band of 1.96·√(0.64·0.36/400) ≈ 0.047, so an observed 0.50 (drop 0.14) is actionable — it clears both a 0.05 degradation threshold and the band — while an observed 0.60 (drop 0.04) is not, even against a hair-trigger threshold, because the band says the data cannot tell. The threshold
Figure 2. Baseline 0.64 with band +/-0.047 (n=400): observed 0.50 (drop 0.14) clears threshold and band; observed 0.60 (drop 0.04) sits inside the band. Worked arithmetic from the frozen monitoring example. trade-off is then arithmetic, not philosophy: a low threshold with a narrow band (large n, stable slices) catches small regressions at the price of false alarms on noise; a high threshold with a wide band (small n, volatile slices) misses real degradation while staying quiet. The chapter fixes no universal threshold because none exists — the declared triplet (threshold, band rule, denominator) is the instrument, and the false-alarm/miss trade-off is priced per slice where the cost of being wrong is known. A drop is actionable only when it clears both the declared degradation threshold and the noise band — small samples produce wide bands, which correctly makes small samples hard to act on rather than easy to overreact to. The chapter claims no universal threshold and no optimality: thresholds are policy, the trade-off between false alarms and missed regressions is explicit, and monitoring only averages is listed as a failure mode the slice identifiers exist to prevent. A dashboard specification falls out directly: one row per (task, slice, period) with correctness, band, incidents, and rights status — the contract’s dashboard requirement satisfied as a record format, not a running service. Concretely, each row carries the candidate identity and model version, the evaluation period, the task and slice identifiers with denominators, the correctness figure with its noise band, the safety incident count (zero by default, never blank), drift indicators against baseline, resource metrics against budget, rights and privacy status re-checked per period rather than copied from promotion time, and an evidence reference back to the manifest or log. Calibration and agreement ride the same monitoring bus as correctness, because a deployed student can hold its accuracy while losing its calibration — confident errors multiplying as the confident-correct shrink — or drift away from its teacher while holding task success, which changes what the next distillation would inherit. The monitoring observation therefore carries the same columns as the evaluation record: correctness, agreement, calibration, cost, and incidents per slice. A drift detection suite that watches only accuracy will certify a model that is simultaneously more wrong-confident and less teacher-faithful than the day it was promoted; the slice identifiers and the multi-column observations exist so that sentence cannot be true unnoticed. Baselines are frozen at promotion time and versioned with the candidate: every window compares against the recorded baseline, never against a moving “recent average” that would normalise a slow bleed into invisibility. A dashboard that cannot show a slice is a dashboard that hides the regression Chapter 35 exists to catch; the slice identifier is therefore mandatory, and any period missing a slice is an ALERT for incomplete monitoring rather than a gap in the chart.
Rollback, simulated completely
The contract requires nine steps; the simulation performs all nine against synthetic post-promotion data. (1) Current approved model: version v2, serving. (2) Candidate evaluation: v3 beats v2 on held-out correctness with all gates passing and human approval recorded — the Scenario A promotion that sets up the fall. (3) Promotion decision: PROMOTE with reasons and evidence on the record. (4) Synthetic post-promotion regression: v3’s correctness on the critical slice drops past threshold and noise band, with two safety incidents logged.
(5) Regression detection: the monitoring window returns ROLLBACK with the breached thresholds named. (6) Rollback decision: restore v2, retire v3 from serving. (7) Restored prior version: v2 serving with its approval record intact — rollback restores, it does not re-promote, so no new approval is manufactured. (8) Incident record: periods, measurements, decisions, and the mandated follow-up, rendered by the same function that will render the real ones. (9) Required follow-up review: post-mortem, re-evaluation of v3, fresh rights review (terms may have changed during the incident window), and no re-promotion without it.
Three properties make this a rollback rather than a reinstall. The prior version was retained with its approval record, so restoration is a decision with evidence, not a redeploy from memory. Canary or staged rollout sharpens the same logic on the way in rather than the way out: a new approval ships to a bounded slice first, with promotion to full serving conditional on the canary windows clearing every threshold — correctness, calibration, incidents, latency, rights — before the old version is retired rather than after.
A canary that cannot be rolled back is not staged deployment but delayed commitment, so the rollback plan is written before the canary ships, naming the restore version, the trigger thresholds, and the incident reference format in advance. The incident log is append-only: the failed v3 episode stays on the record after v3 leaves serving, with periods, measurements, decisions, and the mandated follow-up — the post-mortem asks why the gates did not catch it sooner, and any threshold or coverage change that results is itself a human decision with a reviewer attached, never a silent config edit.
And re-promotion is blocked pending review — the gate treats the rolled-back candidate as a new proposal, because the world that broke it once is admissible evidence about the world that would serve it again. The failure mode this guards is rollback that cannot reconstruct the previous version: weights overwritten in place, configs unversioned, approvals verbal. The workbench answers with boredom — versioned artefacts, hashed configs, recorded approvals — so that “restore v2” names a thing that exists rather than a hope that it does.
Version control of deployed artefacts closes the loop the rollback section opened: every serving version is an immutable, hashed, rights-checked record — weights, config, approval, and the evaluation that promoted it — so that “v2” always names the same bytes with the same permission that put it there. A deployment that mutates weights in place, retrains silently, or serves from an unversioned checkpoint has no previous version to restore and no approval to point at; its rollback plan is fiction, and the version-control check fails it before any incident does. Immutability is not nostalgia for old models — it is the precondition for every other guarantee in this chapter, because a gate cannot govern what it cannot name.
Re-distillation and rights review triggers
Re-distillation is a new experiment, not a refresh: new data means new provenance, new teacher outputs mean new permissions, changed terms mean re-review before collection rather than after training. The triggers are explicit — measured drift past threshold, safety incident, teacher or data licence change, evaluation slice added, performance degradation confirmed — and each trigger names the same sequence: freeze the current approved version, record the trigger with evidence, re-run the rights check for the new action, retrain under a new run identity, re-evaluate on held-out data, re-audit lineage, and decide again with a human.
A worked drift case: the deployed student’s critical-slice correctness falls from 0.64 to 0.55 over two monitoring periods (drop 0.09, band ±0.05 — actionable), no single incident explains it, and the input logs show a shift toward longer inputs the training distribution barely covered. The response is not a hotfix but a re-distillation proposal with a refreshed teaching set covering the new lengths, a new rights check (the longer inputs come from a newly collected slice with its own provenance), and the full gate before the new student can replace the old one. Drift that triggers retraining without re-review is the feedback loop that reinforces errors wearing a monitoring badge.
Feedback loops that reinforce errors get one paragraph of refusal: training on the system’s own unverified outputs without an independent ground-truth check is the E3-preference failure mode worn as a deployment strategy — preference accuracy rising while exact match stays at zero, now with users supplying the preference signal and no oracle in the loop. The gate blocks any re-distillation proposal whose teaching signal lacks independent verification, and monitoring treats a rising proxy metric alongside a flat or falling capability metric as drift until proven otherwise. Improvement without ground truth is not improvement; it is the system teaching itself its own mistakes with increasing confidence. The loop that improves is the loop that re-checks; anything else is drift with a training budget.
Human governance
The system never authorises itself through its own evaluation output. Reserved to qualified humans: rights interpretation, acceptance of residual risks, publication, commercial deployment, incident adjudication, model retirement, unresolved privacy obligations, and changes to performance or safety thresholds. Residual-risk acceptance deserves one paragraph because it is the most abused line in deployment paperwork: accepting a risk means naming it, bounding it, assigning its owner, and recording what would revoke the acceptance — a slice floor with a named exception and a review date is acceptance; a vague “known limitations” paragraph is abdication with better typography. Incident adjudication follows the same discipline: the log records what happened, the post-mortem records why the gates did not catch it sooner, and the threshold or coverage change that results is itself a human decision with a reviewer attached.
Each reservation has the same shape — a named role,
a recorded decision, a date — and the promotion summary’s
human_approval key stays outstanding until all of them that
apply are filled. Threshold changes deserve emphasis because
they are the quietest way to fake improvement: lowering a floor
after a failure is not governance but editing the test, so
threshold revisions require the same human record as promotions,
with the old threshold preserved alongside the new one and the
reason stated. Organisation-specific deployment policy sits
above all of this and is not written here; the chapter provides
the hooks (roles, records, triggers) that such a policy would
hang its decisions on, and stops where counsel begins.
The regulatory backdrop is recorded, not resolved: the AI Act’s post-market monitoring and incident-reporting duties for providers and deployers (L-023, with the Article 113 timetable caveat carried from Chapters 36–38 — verify the consolidated text before stating obligations as fact), logging and record-keeping proportionate to risk, retention bounded by purpose, and re-review when terms, data, or intended use change. Every one of those duties maps onto a workbench record rather than a new process: the monitoring log is the post-market file, the incident record is the reporting input, the lineage graph is the technical documentation’s evidence base. Mapping is not compliance, and the chapter claims none — it claims readiness to be audited, which is a different and checkable thing.
What “continuously improving” would actually require
Five rungs, each strictly harder than the last. Repeated fine-tuning re-runs training — reproduction, not improvement. Continual learning adds new tasks without catastrophic forgetting — a capability claim needing its own evidence, unmeasured in this book. Automated evaluation scores candidates without human hands — telemetry, not judgement. Controlled promotion gates releases on evidence with human approval — this chapter’s machinery, demonstrated in simulation (the PROMOTE path only on synthetic candidates).
A genuinely continuously improving deployed system closes the loop across all four under monitoring, rollback, incident response, and re-review — lawful, auditable, and owned. The book builds the fourth rung and specifies the fifth; it operates no rung above the fourth, authorises none, and calls nothing it built autonomous, self-improving, or deployed. The distance between controlled promotion and a genuinely improving system is measured in incidents survived, not features shipped.
Final conclusion
A student is valuable only when we can establish what it learned, what it failed to learn, what it costs, what risks it introduces, and whether its use is authorised. That sentence opened the mission and closes the book, and every clause now points at machinery rather than aspiration. What it learned: held-out correctness under matched budgets, agreement kept separate, calibration alongside.
What it failed to learn: the zeros and failures preserved in every chapter that measured them, never averaged away. What it costs: parameters, bytes, latency, and memory measured on one host, storage separated from speed. What risks it introduces: slice regressions, confident errors, teacher faults, and unknown permissions, each with a gate that refuses. Whether its use is authorised: action-specific records with named reviewers and dates, and blocked rows that stay blocked until the approvals exist.
The book does not deploy. It does not claim that distillation at frontier scale behaves as it behaved here, that its gates satisfy any regulator, or that its refusals substitute for legal advice. What it claims is smaller and, the author hopes, harder to break: forty-two chapters of measured, dated, reviewable evidence about how behaviour transfers from teacher to student — and a working demonstration that the most important capability in such a system is the ability to say no.
Each clause of the central argument now has an address. What it learned: Chapters 5, 18, 22, 25, 41. What it failed to learn: Chapters 26, 28, 32, 41. What it costs: Chapters 31, 33, 40. What risks it introduces: Chapters 35, 38, 42. Whether its use is authorised: Chapters 36, 37, 39, 42. A student valuable on all five counts is a student worth deploying; this book ships the checklist, not the certificate.
References
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H. and Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM 64(12). arXiv:1803.09010. https://arxiv.org/abs/1803.09010
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D. and Gebru, T. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT 2019).* https://doi.org/10.1145/3287560.3287596
Legal sources
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). In force 1 August 2024; general-purpose AI obligations from 2 August 2025. https://eur-lex.europa.eu/eli/reg/2024/1689/oj (retrieved 12 November 2026). Cited as L-023 for post-market monitoring, incident-reporting, and documentation duties; dates subject to the Article 113 caveat; not legal advice.