Operating, Auditing and Rolling Back a Fine-Tune
Keep experiment completion, evaluation, review, authorisation, activation, health and rollback as separate states, then run a sandbox incident in which a release label stays correct while the loaded model changes, and a retried rollback restores all 24 baseline outputs and the original model identity.
Chapter 41 ended with a service that returns exactly the model it was given. The artifact is identified, the interface is strict and the responses match the library. Every technical precondition for a release is met. None of them answers the question this chapter asks: what distinguishes a technically valid model artifact from an authorised and safely operated release?
The difference is a set of states that are easy to blur together. Someone has to decide that the evaluation was adequate, and someone, perhaps someone else, has to authorise putting the artifact in front of users. After activation, someone has to watch the release and notice when it goes wrong. And someone must be able to put the previous version back, then confirm that what came back is really the previous version and not only its name. Google’s Site Reliability Engineering treats release engineering as a discipline of its own (chapter 8). Google’s reliability engineers state the corresponding operational rule bluntly: there is no good rollout unless a rollback is ready for it.
This chapter builds that discipline at the scale of the book’s synthetic workflow. Its demonstration contains a deliberate trap. A runtime fault replaces the loaded model while the release label stays exactly as it was. A check of the label would pass. A check of the model’s identity fails. The rollback then has to restore behaviour, not just a pointer.
The integrated record is FT-P7-INTEGRATED-01-42-s37. Everything in it happens in memory, in a sandbox, under simulated approval. It is not a production deployment. Its limits are stated in full at the end of the chapter, and they apply to every sentence before them.
States that must stay separate
A model on its way to users passes through a sequence of states. Each state is established by different evidence and owned by a different authority.
| State | What establishes it | Who or what may grant it |
|---|---|---|
| Experiment completed | the run finished and its records are sealed | the experiment harness |
| Evaluation passed | task success meets a gate declared before the run | the evaluator, against the contract |
| Independently reviewed | a reviewer examined the evidence and its limitations | a person or process separate from the author |
| Release authorised | an explicit decision to expose the artifact | an authority separate from the artifact |
| Activated | the artifact is the one actually serving | the deployment controller |
| Healthy | the serving model’s actual identity matches the release | a health check on runtime state |
| Monitored | observed behaviour stays within declared limits | monitoring over real or recorded events |
| Rolled back | a previous authorised release serves again | the controller, then verified by health |
Two confusions cause most of the damage. The first treats a passed evaluation as permission to release. An evaluation is evidence for a decision, not the decision, and Chapters 37 to 39 showed how much an evaluation can miss. The second treats an artifact’s own fields as its approval. If a manifest could say “approved: true” and be believed, anyone who can write a manifest could approve a release.
The figure keeps the eight states and their separate authorities in view, with the sandbox trap alongside: the v2 label stays correct while the loaded model changes, so only the stored-identity health check fails. It is a sandbox diagram with simulated approval, not a production deployment.
The release manifest
The sandbox registers each candidate with a sealed release manifest. Its fields fall into groups.
- What it is: version, base identity, adapter identity, model identity, export-manifest identity and the serving configuration (loopback host, 4 concurrent inferences, FP64).
- What it was built and judged on: task contract, dataset digest, a configuration note (“Deterministic constructed adapters, no training”), the evaluation digest, the test results, the source digest and the environment digest.
- What may be done with it: rights status (
NEW_SYNTHETIC_EDUCATIONAL_ONLY), known limitations, scope (SYNTHETIC_SANDBOX_ONLY) and rollback target. - Where it stands: experiment state
COMPLETED, evaluation statePASSED, review statePENDINGandrelease_authorised = false.
The last group is the important one. At registration, the manifest must
say that review is pending and that release is not authorised. A candidate that
arrives claiming its own approval is refused with FORGED_APPROVAL. Approval is
recorded outside the artifact, in the controller’s state and audit log, and can
be granted only by a reviewer object that the sandbox was constructed with and
that no artifact can supply. In this experiment that reviewer is an in-process
capability token. Holding it simulates a separate authority. It does not
authenticate anyone.
Registration checks internal consistency before anything else. The candidate must belong to the synthetic sandbox scope and pass the 0.9 task-success gate declared before the run. It must carry the synthetic-only rights status and the bounded serving configuration. Its model and adapter identities must match both the loaded model and the export manifest, its export-manifest digest must match, and its version must be new, because versions are immutable. All of these checks show that the artifact is what it says it is. None of them shows that anyone should use it. Internal consistency and authenticated approval are different properties, and the manifest can only provide the first.
The demonstration
Two candidates were built from the constructed adapters of Chapter 37:
- v1 is the combined system with adapter weights (1, 1);
- v2 is a candidate with weights (1, 1.25), a slightly stronger review adapter.
Each was exported, reloaded through Chapter 41’s identity chain, and evaluated on the 24 test cases. Both scored task success 1.0 on 24 cases, above the 0.9 gate. Each was registered, then approved by the simulation controller. The sequence then ran as follows.
- A dry run of v2 confirmed that it would activate.
- v1 was activated, then v2 was activated. Health before the fault: v2, healthy, with its actual model identity equal to its release identity.
- Controlled runtime regression. The model object registered under v2 was replaced in memory with a system containing adapter A only. The release manifest, the export files and the active label were all left unchanged.
- Health check. Active version still v2. Expected model identity
4cfaefa8…, actual4d1dfd72…. Healthy: false. - Regressed behaviour. The sandbox’s outputs on the 24 test cases now
differed from the v1 baseline in 12 of 24. Every review case was answered
with
completeorrefuse, exactly as adapter A alone answered them in Chapter 37. - Monitoring. Three recorded synthetic events were evaluated against the declared contract, and the rollback trigger fired.
- Interrupted rollback. A rollback to v1 was prepared, then interrupted before commit. Result: not committed, active version still v2.
- Retried rollback. The same rollback was run again. Result: committed,
active v1, health true, actual identity
d3f254e1…equal to v1’s release identity. Reported as not idempotent, because the active version changed. - Restored behaviour. The sandbox’s outputs on the 24 test cases were compared with v1’s baseline outputs. 24 of 24 were identical.
- Repeated rollback. Rolling back to v1 again committed, left v1 active and healthy, and was reported as idempotent, because nothing needed to change.
The authoring cross-check verified each of these steps from the raw record without using the library. It rebuilt the baseline outputs from the v1 weights and the regressed outputs from adapter A’s weights. It confirmed that the regressed model’s identity is the A-only identity recorded in Chapter 39’s invalid-output control. It also confirmed the restored outputs’ equality case by case.
Why the label was not enough
At step 4 the release label said v2, the export files for v2 were intact and their hashes would have verified, and the release manifest was unchanged. Every check that looks at names or files would have passed. The model actually answering requests was a different model. It had the wrong identity and lost half the task.
Real systems reach this state in mundane ways: a weight file overwritten in place, an adapter loaded by name from the wrong directory, a cache serving an older object, a hot-reload that half-completed. The defence is the one the sandbox uses. Health compares the runtime object’s stored identity with the identity the release manifest promised. The identity was constructed from declared configuration when the object was created; this check does not rehash live tensors. It detects the substituted object in this demonstration, but would not detect an in-place tensor mutation that left the stored identity unchanged. A health check that reports the configured version reports configuration, not health.
The same principle applies to rollback. Step 8 is not complete when the pointer moves to v1. It is complete when the restored model’s identity matches v1’s release and its outputs match v1’s recorded behaviour. A rollback that changes the label and leaves the faulty object in place has failed, whatever the label says.
Rollback as a protocol
The sandbox’s rollback has four phases, each recorded.
stateDiagram-v2
[*] --> Prepared: target registered, authorised, identity intact
Prepared --> Committed: commit pointer
Prepared --> Interrupted: interruption before commit
Interrupted --> Prepared: retry
Committed --> Verified: recompute health
Verified --> Recorded: ROLLED_BACK + idempotence flag
Verified --> HealthFailed: not healthy
Recorded --> [*]
- Prepare. Check that the target is registered, authorised and still has its
promised identity, then record
ROLLBACK_PREPAREDwith the source and target versions. - Commit. Switch the active pointer to the target.
- Verify. Recompute health on the newly active model. If it is not healthy,
raise
ROLLBACK_HEALTHinstead of reporting success. - Record. Append
ROLLED_BACKwith an idempotence flag.
An interruption before step 2 leaves the active pointer where it was. The
experiment injected exactly that interruption, observed committed = false with
v2 still active, and recorded ROLLBACK_INTERRUPTED. Idempotence means that
applying the operation again after success changes nothing. The repeated
rollback confirmed it: committed, still v1, flagged idempotent.
This is atomicity in memory, in a single process, for a pointer assignment. It says nothing about what happens if the process dies between steps, if the machine loses power, or if the state is spread across several servers. Those require a durable journal written before each transition, recovery logic that replays or reverses incomplete transitions, and tests that kill the process at every step. None of that was built or tested here.
The audit trail
Every state change appended an event to an audit log. Each event is sealed with
a SHA-256 digest of its contents, and each event’s prior field holds the digest
of the event before it. The demonstration produced 14 events:
| Sequence | Event | Detail |
|---|---|---|
| 0 | CANDIDATE_REGISTERED | v1 |
| 1 | REVIEW_APPROVED | v1, simulated, not production |
| 2 | RELEASE_AUTHORISED | v1, simulated, not production |
| 3 | CANDIDATE_REGISTERED | v2 |
| 4 | REVIEW_APPROVED | v2, simulated, not production |
| 5 | RELEASE_AUTHORISED | v2, simulated, not production |
| 6 | SANDBOX_ACTIVE | v1 |
| 7 | SANDBOX_ACTIVE | v2 |
| 8 | ROLLBACK_PREPARED | v2 → v1 |
| 9 | ROLLBACK_INTERRUPTED | active v2, target v1 |
| 10 | ROLLBACK_PREPARED | v2 → v1 |
| 11 | ROLLED_BACK | v1, not idempotent |
| 12 | ROLLBACK_PREPARED | v1 → v1 |
| 13 | ROLLED_BACK | v1, idempotent |
The cross-check recomputed every seal and every link. The chain is intact.
A hash chain makes undetected editing of the middle of the log harder. Changing one event changes its digest and breaks the next event’s link. It does not stop someone who controls the log from rewriting the whole chain, truncating its tail or never writing an event at all. A tamper-evident audit log for real releases needs its head anchored somewhere the writer cannot change, such as a separate service, a signed checkpoint or an append-only store under different control, and its events authenticated by whoever performed them. Note also what the log does not contain: the runtime substitution at step 3. A fault that bypasses the controller leaves no audit event. It was detected by the health check, not by the log. Logs record what the controller did, and health checks reveal what the system is actually doing.
Monitoring and drift
The monitoring contract declares, for every observed request: success, schema validity, latency, abstention, policy violation, adapter identity and an input-shift flag. A request counts as a failure if it was unsuccessful, schema-invalid, a policy violation, or served by an adapter set other than the release’s. The contract also declares a latency limit of 0.05 s and an error budget of 0.1, a failure rate above which rollback is triggered.
The experiment fed the contract three recorded synthetic events, each
describing an unsuccessful, schema-valid request served by the A-only adapter
set, with a recorded latency of 0.01 s. The result was 3 failures out of 3, a
failure rate of 1.0, 0 abstentions, 0 shift flags, and a rollback trigger with
two reasons: ADAPTER_IDENTITY and ERROR_BUDGET.
Read that precisely. The events were written into the experiment, not observed from traffic. The 0.01 s latencies are example values, not measurements of any service. The demonstration shows that the monitoring arithmetic and trigger logic behave as declared on a known input. It does not show that anything was monitored. Continuous monitoring would require collecting events from real requests, labelling success, which needs ground truth that production traffic rarely provides at once, detecting input drift statistically, and alerting people who can act. Testing and monitoring requirements of that kind are catalogued in the ML Test Score rubric (Breck et al., 2017). The broader cost of neglecting them is the subject of Sculley et al. (2015).
What the experiment shows and does not show
It shows that the release states can be kept separate in code. A candidate cannot approve itself, approval comes from a distinct capability, and every transition is recorded in a linked log. It shows that a health check based on the runtime object’s identity detects a fault that every label-based and file-based check would miss. And it shows that an interrupted rollback leaves state unchanged, that a retried one restores the original identity and all 24 baseline outputs, and that a repeated one is idempotent.
It does not show a production deployment, which never occurred. It does not show
real review or authorisation, which were simulated by an in-process token. It
does not show crash or power-loss recovery, since rollback lived in one process’s
memory with no durable journal. It does not show continuously authenticated
model state, since health compares stored identity fields when asked; it does
not rehash live tensors or provide cryptographic attestation. And it does not show monitoring of real traffic,
because the three events were synthetic. The record states
AUTHENTICATED_EXECUTION = NOT_ESTABLISHED.
Practical lessons
- Model releases as explicit states, and never let one imply another. An evaluation pass is not a review, a review is not an authorisation, and authorisation is not health.
- Keep approval outside the artifact. A manifest field that grants permission is a vulnerability.
- Check the identity of the object actually serving. This sandbox compares stored identity fields; stronger health checks must verify live model state.
- Treat rollback as a protocol: prepare, commit, verify and record. Verify restored behaviour as well as restored identity.
- Make rollback idempotent, and test interruption at every step. A rollback that survives only an interruption in memory is not crash-safe.
- Anchor audit logs outside the writer’s control, and remember that faults bypassing the controller never reach the log.
- Declare monitoring contracts, error budgets and rollback triggers before activation, and distinguish recorded test events from observed traffic.
Limitations
This is an in-memory sandbox in one process. Review and authorisation are simulated with a capability token, not performed by people or authenticated systems. Rollback has no durable journal and was not tested against process death, power loss or distributed state. The regression was a deliberate object substitution, not an observed failure. Monitoring used three synthetic events with example latencies. Only one data seed was run. Both candidates are constructed models on the finite synthetic task of Chapter 37. The audit chain is local and unanchored.
FineTuneLab
ftlab.delivery.operations provides release, which builds a sealed release
manifest from an export manifest and a matching evaluation, and
validate_candidate, which enforces scope, gate, rights, serving bounds and the
pending-review requirement. It also provides Sandbox, an in-memory registry
with simulated approval, dry runs, activation, identity-based health, interrupted
and repeated rollback, and a hash-chained audit log, and monitor, which applies
the declared event contract. The experiment is ch42 in
ftlab.delivery.experiments. The module exposes no production deployment
interface.
The argument of Part VII
Part VII followed one small adapted system through six stages, and each stage found a way for a correct-looking result to mislead.
Measure. Chapter 37 separated the model’s probabilities from the system’s delivered output. A postprocessor fault lowered task success to 23/24 while the model’s negative log-likelihood stayed at 0.284779. Model evaluation and system evaluation are different measurements.
Compare. Chapter 38 put intervals around differences. The same 16/24 effect produced a zero-width interval at one data seed and [10/24, 20/24] at the other. Data-seed, bootstrap-seed and training-seed variation are different things, and an interval describes its procedure, not the world.
Challenge. Chapter 39 linked 144 challenges to 24 origins. It distinguished inputs that only look new from patterns that are truly unseen, and it traced a 0/24 score under a policy revision to a stale policy source, not to collapsed action selection.
Compose. Chapter 40 derived adapter composition as weight arithmetic. It confirmed exact merge equivalence for this tiny FP64 model and showed a compatible but conflicting delta erasing a retained capability.
Serve. Chapter 41 exported, reloaded and served the composed model, with 24/24 responses matching the library. It showed that every timing figure belongs to its own boundary.
Operate. This chapter showed that a release label can be right while the model is wrong, and that rollback must restore behaviour as well as names.
Put together, these chapters make one claim. A model can pass an evaluation, fail under shift, produce a correct export, suffer a system-level regression and require rollback, all without any single measurement being wrong. Every number in Part VII was correct for its boundary. The failures lived between the boundaries: between model and postprocessor, between in-distribution and shifted inputs, between policy and citation, between component and composition, between library and service, and between label and loaded object. The integrity of a fine-tuned system is the integrity of every link in that chain, and each link has to be measured where it actually is.
Throughout, the perfect rule baseline stood beside every adapter result. On a finite task defined by rules, the rules win. That was never the point of the experiments. The point was to build and test the machinery that tells the difference between a system that works and one that only appears to.
Where this leads
Part VIII, Specialisation and Capstone, brings the whole of FineTuneLab together. Its first chapters move beyond this deliberately small synthetic workflow, to multilingual and multi-task adaptation, to tool use and structured outputs, and to long contexts, multimodal models and mixtures of experts. Its last three chapters integrate the FineTuneLab workflow into a usable solution, run and compare its training pipeline, and release it with a runbook. That covers data, training, evaluation, comparison, challenge, composition, serving and operation. Every stage there will be held to the boundaries that Part VII drew.
Further Reading
- David McNutt, “Release Engineering” (2016). Read the source. The release discipline behind hermetic builds, staged rollouts and rollback readiness that the sandbox only rehearses.
- Eric Breck et al., “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction” (2017). Read the source. The production-readiness and monitoring rubric the chapter’s monitoring contract is measured against.
- Saleema Amershi et al., “Software Engineering for Machine Learning: A Case Study” (2019). Read the source. Observed ML workflows showing why data, entanglement and versioning dominate operating cost.