Chapter 18 ended with an information boundary we could finally audit: the firewall blocked all 288 constructed attacks in its deterministic train/dev suite. That does not make the downstream metric correct; it means the tested result was not produced by the forbidden evidence paths we attacked. Chapter 18 also said the obvious next thing — even an admissible offline result is not a production version.

This chapter closes two gaps between those two facts.

The first is a debt. Chapter 7 sealed a holdout — seven cases across two source families — and promised to spend it exactly once, here, on whichever program the process selected, and to report the result whether or not it flattered the preceding chapters. This chapter pays that debt in section 1.

The second is the mechanism. Between a spent holdout and an active deployment sits a sequence of separate decisions — independent evaluation, comparison, promotion, activation, rollback — each with its own record and its own authority. Collapsing them into optimizer.compile(...) → production is how a good optimizer score becomes an outage.

    flowchart LR
    CA[candidate artifact] --> IE[independent evaluation]
    IE --> CM[comparison]
    CM --> PD[promotion decision]
    PD --> DM[deployment manifest]
    DM --> AR[activation record]
    AR --> OB[observation]
    OB --> RB[rollback record]
  

Each box is a separate record with its own authority. None of that is a DSPy feature. Promotion is a governance boundary, and the surrounding system owns it.

1. Spending the holdout, once

The holdout-selection decision uses the three optimizer results produced under the frozen editorial v1 protocol in Chapters 11 through 13. Bootstrap’s paired gain was +0.0003; MIPRO’s was +0.0199, the largest of the three; and GEPA accepted no child. None cleared the configured promotion margin of +0.05.

Chapter 14 is deliberately not a fourth candidate in that comparison. It changes the objective. Its GEPA result of +0.0909 is development-set selection evidence on a different scoring scale and protocol, so feeding it into the editorial-v1 promotion decision would violate the comparability rule this chapter is about.

So the program the process selects for the one holdout evaluation is the one the book started with: the frozen baseline. spend_holdout.py makes that selection mechanically — it reads the recorded development runs for the three optimizers, checks whether any candidate beat the baseline dev mean of 0.7992 by the 0.05 margin (the closest, MIPRO’s, fell three hundredths short), finds none, and loads EditorialRewriteProgram() unmodified.

Then it evaluates the sealed families once. The holdout is technical-explanation and email-correspondence — seven cases, ed-004 and ed-039 through ed-044. No optimizer used these cases during candidate construction or selection, and their reference rewrites, semantic outcomes, and per-case evaluation results remained sealed until this final spend.

Seven of seven cases produced a valid rewrite. Zero semantic violations — v2 equals v1 on every case, so the LM judge flagged nothing. The weakest case was ed-004 at 0.683, where the baseline clarified the cache risk but padded the sentence with “posing a technical risk for developers”; the strongest were ed-040 and ed-044 near 0.97.

Now sit with the number. The frozen baseline scored 0.8693 on the sealed holdout and about 0.80 on the development family it had been studied against since Chapter 8. In this one seven-case spend, the holdout score is higher. That does not establish that the holdout families are intrinsically easier; the family mix changed and this is one session, not an estimate of a difficulty distribution.

The other fact is firmer: roughly 189,000 tokens of optimizer search under the earlier editorial objective produced no candidate that qualified to reach this evaluation. Chapter 14 later showed that changing the objective changes the search result, but that is a different protocol.

This is the result Chapter 7 promised to report regardless. It does not say optimization is worthless. It says that under the frozen editorial-v1 selection protocol used for this holdout spend, the honest final artifact is the program we did not change. Its 0.8693 is a favourable observation on these seven sealed cases, not a generalisation rate.

2. “Spend once” is a mechanism, not a slogan

Chapter 18 argued that a held-out set stops being independent evidence once its outcomes influence another design or selection decision. spend_holdout.py makes the intended one-shot protocol difficult to violate accidentally:

• The final candidate is chosen from development evidence only. The holdout plays no part in selection.
• The holdout is evaluated once, in one pass, and the result is written to holdout_result.json.
• On any second invocation, the script prints REFUSING: holdout already spent and exits non-zero.
• The evaluation contains no adaptation loop that reads a per-case holdout result and changes the program.

This chapter does inspect the final artifact in order to report it. That is why we can say ed-004 scored 0.683 and describe what its rewrite did. The distinction is temporal: model selection was frozen before that inspection, and no subsequent candidate is allowed to use what we learned.

After this report, these seven outcomes are burned as independent evidence for this experimental cycle. Deleting holdout_result.json can override the mechanical guard; it cannot make the outcomes unknown again.

The refusal is therefore a safeguard, not magic. If these results guide another tweak, the next independent evaluation needs fresh evidence.

3. An offline win is not an activation

The holdout result is evidence. What that evidence is allowed to do is a separate question, and the answer is not “the higher number wins.”

A candidate can beat the aggregate and still be unfit for activation:

it introduced a hard regression on a case the baseline handled
it increased invalid outputs
a minority family got worse while the mean went up
the evaluation ran under a different protocol than the baseline's
it was compiled against a baseline that is no longer active
there were too few independent cases to trust the delta

The governance experiment for this chapter, experiments/dspy-from-first-principles/ch17_promotion, encodes the comparison as an ordered sequence of gates, not a weighted blend. Transcribed from compare(), the precedence is:

1  candidate's baseline fingerprint == the active baseline?     no  → INSUFFICIENT (stale lineage)
2  candidate's evaluation protocol == the active's?             no  → INSUFFICIENT (protocol mismatch)
3  candidate evaluated on at least min_cases?                    no  → INSUFFICIENT (too few cases)
4  candidate.hard_failures > active.hard_failures?              yes  → REJECT (hard regression)
5  candidate.validation_failures > active.validation_failures?  yes  → REJECT (validation regression)
6  candidate.score >= active.score + min_improvement?           yes  → PROMOTE
7  otherwise                                                         → INSUFFICIENT (below threshold)

The aggregate score is checked at step 6 — after lineage, protocol, evidence sufficiency, and both regression gates. A candidate scoring 0.95 with a hard regression never reaches step 6. This is CoCoder’s lesson in policy form: it compares dimensions — completed executions, invalid outputs, scope rejections, validation pass/fail, material regressions — and rejects any candidate that increases hard regressions or reduces validation passes, whatever the headline number does.

4. Seven scenarios

The runner exercises seven decision scenarios spanning PROMOTE, REJECT, and four INSUFFICIENT outcomes. It does not separately exercise the source branch where the active baseline fingerprint is stale; the stale-lineage scenario below exercises the candidate-stale branch. So this is broad policy-path coverage, not complete branch coverage of compare().

The active baseline is editorial-rewrite-v1: synthetic score 0.72, three cases, zero hard failures, zero validation failures. The policy requires min_cases = 2 and min_improvement = 0.05.

Decision counts: one PROMOTE, two REJECT, four INSUFFICIENT.

Five separate 0.95 candidates — a score well above the baseline’s 0.72 and far past the 0.05 margin — did not activate. Two were rejected for regressions the aggregate could not see. Three were ruled insufficient: too few independent cases, a mismatched evaluation protocol, and a stale baseline lineage. The one candidate that promoted scored 0.81, nine points over baseline, with nothing broken.

The 0.75 candidate is the quiet one. In this synthetic scenario, it has no hard or validation regressions and its constructed aggregate is +0.03 above the active program. The policy still returns INSUFFICIENT because +0.03 is below the configured +0.05 margin.

That demonstrates threshold behavior. It does not establish that +0.03 is a statistically real gain, or that +0.05 is calibrated to a universal noise level. Section 10 is where those synthetic inputs and policy choices are bounded explicitly.

5. REJECT and INSUFFICIENT are different answers

The runner keeps these as separate decision classes on purpose, and the distinction carries real information:

• REJECT means the evidence contains a disqualifying fact. The candidate introduced a hard or validation regression. Do not promote this candidate.
• INSUFFICIENT means the evidence does not justify activation. The candidate might be fine — the protocol drifted, the lineage is stale, there were too few cases, the gain was too small. Get better evidence.

Collapsing both into “not promoted” throws away the reason. REJECT says the available evidence contains a disqualifying regression under this policy. INSUFFICIENT says the comparison cannot authorize promotion, but the remedy depends on the gate that fired: collect more cases, rerun under a comparable protocol, rebuild against the current baseline, or simply leave a below-threshold candidate inactive.

The four INSUFFICIENT scenarios in the matrix are therefore not four demonstrated bad candidates. Three even carry the synthetic score 0.95; what is missing is admissible promotion evidence.

6. Promotion is an executable gate

A PROMOTE decision is not advisory metadata that a deployment script is trusted to honour. In the runner, activate_after_promotion refuses to do anything unless it is handed an actual PROMOTE record:

if promotion_decision["decision"] != "PROMOTE":
    raise RuntimeError("activation attempted without PROMOTE decision")
if previous_manifest.status != "eligible":
    raise RuntimeError("previous active manifest is not rollback-eligible")
if candidate_manifest.status != "candidate":
    raise RuntimeError("candidate manifest is not in candidate state")

The experiment probes this directly. After promoting the 0.81 candidate, it takes the rejected hard-regression candidate and tries to activate it. The call raises, non_promote_activation_blocked is recorded true, and the run’s invariants only pass because the refusal happened.

Activation is also a state transition with preconditions, not a boolean flip. The previous version must be in eligible state (so it can be rolled back to); the candidate must still be a candidate (not already superseded). Staged rollout — shadow the candidate against live traffic, then a bounded canary, then full activation — is the same idea applied over time, and DSPy provides none of it. The registry that decides which program answers a request is yours.

7. Five records, five fingerprints

The successful path produces five distinct record types, and the run fingerprints each of them:

8. Rollback is part of lineage

If a system can promote, it must be able to reverse a promotion — and reversal has to be rehearsed, not just named.

The runner walks a real state sequence:

    stateDiagram-v2
    [*] --> v1_active
    v1_active --> v1_eligible: v2 enters as candidate
    v1_eligible --> v2_active: PROMOTE
    v2_active --> v1_active: post-activation hard regression, rollback
  

rollback(history) scans the deployment history backwards for the most recent eligible manifest and restores it, producing a separate RollbackRecord with status = "rolled_back" and its own fingerprint. CoCoder does the same thing in its lineage: rollback requires a target that was historically active or superseded, flips the current active version to rolled_back, restores the target to active, and writes a promotion record with decision = "rolled_back". Rollback is a tracked lineage event, not an invisible database edit.

The trap is subtler than “we forgot to keep the old artifact.” An old candidate state that loads is not necessarily the old system. If the provider, retrieval index, tool schema, or validation dependency has drifted since that version was active, loading it may restore the old program state without restoring the old behavior.

A production rollback target therefore needs to pin those runtime dependencies too. The deterministic runner in this chapter does not test that reconstruction: its manifest contains only deployment id, program id, candidate fingerprint, and status. What the rehearsal establishes is lineage and state-transition behavior — selecting an eligible predecessor, restoring it to active, and recording the reversal — not full behavioral restoration across runtime drift.

9. Production output is not a training label

After activation, every request should produce a record even when nothing dramatic happens — request id, program version, model and provider, fallback state, validation status, failure reason, input and output fingerprints, trace id. For repository repair, add tool calls, changed files, test targets, and scope result; for editorial work, add the evidence packet hash and any review decision.

That record is not immediately training data. Two inferences are tempting and wrong:

user did not complain          ≠   positive label
human preferred the candidate  ≠   the candidate was applied and it worked

Writer’s architecture keeps these apart — preference evidence, candidate-generation evidence, blind review, application evidence, and later ranker statistics are separate streams. This book adds the stricter rule from Chapter 18: a production observation becomes candidate future evidence only after it matures, keeps its provenance, passes the leakage policy, and is put into decision-time form. Until then it is an event, not a label.

10. What this establishes, and what it does not

The holdout spend is one program, seven cases, one session. 0.8693 with zero recorded semantic violations is the book’s one sealed editorial holdout measurement. It is not a distribution, and the fact that it exceeds the development score does not establish that the holdout families are intrinsically easier. Seven cases across two families establish only what happened in this spend. After this chapter inspects and reports the per-case outcomes, those outcomes are no longer fresh independent evidence for a future iteration.

The governance run is deterministic and synthetic. The candidate “scores” in the seven scenarios are hand-constructed inputs chosen to exercise selected policy paths, not measured model outputs. The run proves those paths return the expected decisions. It does not provide complete branch coverage of compare(), and it does not prove that min_cases = 2 or min_improvement = 0.05 are statistically calibrated thresholds for any deployment.

Activation and rollback were rehearsed on in-memory teaching manifests. No service was running, and the measured manifest does not pin provider, model, index, tool, or validation versions. The run establishes the state-machine properties it actually exercises: a non-PROMOTE decision cannot activate, activation has manifest-state preconditions, rollback selects an eligible predecessor, and each transition produces a separate record. Reconstructing the same behavior under real runtime drift remains production work.

What survives all of those limits is the structure:

Independent evaluation, comparison, promotion, activation, and rollback are separate authorities with separate records. An optimizer score can be evidence for comparison; it has no authority to activate a program by itself.

What Usually Goes Wrong

Conclusion

We spent the sealed editorial holdout exactly once. Under the frozen v1 selection protocol, the comparable optimizer results from Chapters 11 through 13 produced no candidate that cleared the configured 0.05 promotion margin, so the process selected the frozen baseline. It scored 0.8693 on the seven held-out cases, higher than its development measurement, with no recorded semantic violations. Chapter 14’s alternate-objective result is deliberately outside that comparison because it changes the evaluation protocol.

Then we built the boundary that decides what such evidence is allowed to do. compare() checks lineage, protocol comparability, case sufficiency, and regressions before it checks aggregate improvement. The seven synthetic scenarios produced one PROMOTE, two REJECT, and four INSUFFICIENT, while the activation path raised when handed a non-PROMOTE decision. Candidate identity, promotion decision, deployment manifest, activation record, and rollback record remain separate fingerprinted artifacts. The measured rollback is a lineage/state-machine rehearsal; pinning the full runtime is the additional requirement a production manifest must satisfy.

We removed the assumption that an optimized program should serve real work automatically.

Every mechanism the book needs now exists on its own. The final chapter asks whether they compose into a single loop.

Further Reading

• The Reusable Holdout: Preserving Validity in Adaptive Data Analysis (Dwork et al., Science 2015): why repeatedly consulting a held-out set — even indirectly, through decisions it informs — destroys the guarantee it was supposed to provide. The mechanical case for spending once. (Science 349(6248))
• Hidden Technical Debt in Machine Learning Systems (Sculley et al., 2015): the model is a small box in a large system of configuration, monitoring, and glue — the gap this chapter’s manifest and lifecycle are trying to close. (NeurIPS 2015)
• The ML Test Score: A Rubric for ML Production Readiness (Breck et al., 2017): a concrete checklist covering evaluation, monitoring, and the ability to roll back — the operational side of promotion. (IEEE Big Data 2017)