From Experiment to Production
From Experiment to Production
Chapter 16 gave us a candidate that can be evaluated without obvious leakage.
That still does not mean:
optimizer.compile(...)
↓
production
The lifecycle is closer to:
1. Offline win is insufficient
A candidate may beat an aggregate metric and still be unacceptable:
2. CoCoder’s promotion boundary
CoCoder stores separate records:
ProgramEvaluationCaseResultDTO
ProgramEvaluationDTO
ProgramComparisonDTO
ProgramPromotionDTO
The docs summarize the invariant:
optimizer proposes
CoCoder evaluates
CoCoder compares
operator/system policy promotes
Promotion requires a persisted comparison, a promote recommendation unless there is an explicit override, the comparison baseline still being the single active version, the target still being a candidate, and optimizer lineage—when present—matching that active baseline.
An override changes the governance decision; it does not erase the underlying comparison evidence. Both should remain queryable.
In CoCoder, promote(...) then performs the status transition transactionally:
3. DSPy artifact vs deployment manifest
DSPy can save compiled parameter state:
4. Service boundary
A minimal service shape:
5. Observability is part of activation
A production run should produce a record even when nothing dramatic happens. Runtime identity should come from the deployment/runtime boundary, not from fields the model happened to return.
For repository repair, add tool-call summaries, repository revision, changed files, test targets, validation artifact IDs, scope result, and isolated-workspace identity. For editorial work, add candidate source, evidence-packet fingerprint, review decision, application status, and any later outcome references.
Do not put every raw input, tool observation, or model trace into one analytics row by default. Production records may contain sensitive or high-volume material. Keep durable identity/fingerprints in the run record and store larger evidence under an explicit retention and access policy.
The record is not automatically a training example. It becomes source material for a future controlled dataset only after:
6. Shadow, canary, activation
Safe activation is staged.
Shadow:
7. Production feedback is not automatically a label
Be careful:
program produced result
↓
user did not complain
≠
positive label
Also:
human preferred candidate
↓
candidate was not applied
≠
successful production outcome
Writer’s architecture distinguishes preference evidence, candidate-generation evidence, blind review, application evidence, and later statistics/ranker evaluation. That separation illustrates a broader production rule:
8. Rollback is a feature, not a failure
If a system can activate a version, it must be able to reverse that activation:
What Usually Goes Wrong
| Symptom | Likely cause | How to diagnose it | What to change |
|---|---|---|---|
| Candidate wins offline and breaks production | Offline/online execution contract differs | Compare deployment fingerprint, LM/adapter, tools/indexes, traffic slices, latency, and fallback behavior | Fix parity first; shadow and canary before full activation |
| Shadow run changes production state | Candidate has side-effecting tools despite hidden output | Audit tool calls and external mutations | Use read-only or isolated shadow dependencies |
| Cannot explain active behavior | Active registry points to a version without an immutable deployment manifest | Resolve a production request back through run → deployment → promotion/comparison evidence | Require manifest fingerprint on every active deployment and run |
| Run claims the wrong model/version | Runtime provenance came from model output or mutable global config | Compare run record with deployment snapshot and provider telemetry | Resolve identity at the runtime boundary per request |
| Canary result is hard to interpret | Assignment or stop thresholds changed during the run | Audit routing history and gate versions | Freeze assignment and rollback policy before canary start |
| Rollback restores code but still fails | Historical external dependencies drifted or disappeared | Reconstruct the full rollback bundle in staging | Pin/version dependencies and continuously test rollback readiness |
| Optimizer score becomes deployment policy | Optimization, comparison, and activation are fused | Trace the active-version transition | Require persisted independent evaluation, comparison, and promotion decision |
| Production feedback corrupts training data | Observation was converted directly into a label or leaked later outcome into generation view | Audit label lineage and dataset-building policy | Separate observation, mature outcome, derived label, and decision-time training view |
Conclusion
We gained a production lifecycle for DSPy programs: immutable candidate state, independent evaluation, deterministic comparison, explicit promotion, deployment reconstruction checks, controlled activation, attributable production runs, staged exposure, mature feedback, and rollback.
We removed three assumptions: that an optimized program should serve real work automatically, that a saved DSPy state is the same thing as a deployment, and that rollback means loading an old prompt file.
The full lifecycle is now visible:
