Optimize the Instructions
Optimize the Instructions
Chapter 11 let the optimizer change predictor behavior through bootstrapped demonstrations. That changes the examples shown to the LM, but the predictor instructions remain mostly fixed.
Now we expand the allowed mutation surface:
1. What MIPROv2 searches
Current MIPROv2 builds candidate few-shot sets, proposes instruction candidates for predictors, and searches combinations against validation evidence. Its current implementation uses an Optuna-backed search over prompt parameters.
Conceptually:
2. A MIPROv2 compile shape
Current docs show MIPROv2(metric=..., auto="light" | "medium" | "heavy"). Its compile method accepts trainset and valset; internally, the optimizer validates those datasets, builds an evaluator over the validation set, and searches prompt parameters against that development evidence.
For the teaching experiment:
The demo settings are deliberate. As Chapter 11 established, the current teaching rows do not contain stage-compatible labels for the composed program’s leaf predictors, so this experiment disables labeled demos and lets MIPROv2 build few-shot candidates from bootstrapped traces instead.
metric_threshold=0.70 governs acceptance of those bootstrapped traces. It is part of optimizer configuration and must remain frozen during the run.
Passing valset=devset explicitly is also important. Current MIPROv2 can derive a validation set from the trainset when valset is omitted; that would change our predeclared split roles.
The experiment boundary is therefore:
3. Search is not proof
MIPROv2 may find a candidate that scores best on the development data it searched over. That is useful. It is not proof that the candidate is better generally.
optimizer found candidate
≠
candidate is better
The candidate still needs evaluation on cases that did not drive the search.
baseline = EditorialRewriteProgram()
fewshot = compile_few_shot_candidate(trainset)
mipro = compile_mipro_candidate(trainset, devset)
holdout_evaluator = dspy.Evaluate(devset=holdout, metric=editorial_metric)
baseline_holdout = holdout_evaluator(baseline)
fewshot_holdout = holdout_evaluator(fewshot)
mipro_holdout = holdout_evaluator(mipro)
This is the point where an independent comparison can be made, but the teaching corpus still does not support a broad performance claim. It contains only one holdout case. A win on that case is one observed outcome, not evidence that MIPROv2 generally improves editorial rewriting.
This chapter therefore reports no measured result. A serious experiment would need enough holdout cases to expose a failure distribution and enough repeated evidence to distinguish a robust improvement from ordinary variation.
CoCoder’s frozen experiment design follows the same rule at production scale. It freezes the manifest, evaluates the baseline under the frozen holdout protocol, optimizes on allowed train/development data, persists a candidate, audits candidate integrity, evaluates the candidate under the same holdout protocol, compares, persists the result, and stops without automatic promotion.
4. Why development data is different from holdout
MIPROv2 repeatedly evaluates candidate program states against its validation set. Those examples are therefore optimizer-visible development evidence. They help determine which instruction/demo combination survives.
That is not contamination. It is the purpose of development data.
5. Budget is part of the experiment
Instruction search costs LM calls. More exploration can help, but it is not a guarantee of better holdout behavior.
more candidate exploration
↓
more LM calls / time / cost
↓
possibly better dev search
↓
not guaranteed better holdout behavior
Record budget fields:
6. Compare the three stages
By the end of this chapter, the protocol can compare three program states:
| Program | What changed? |
|---|---|
| Baseline | Hand-designed DSPy parameter state |
| BootstrapFewShot candidate | Bootstrapped predictor demonstrations |
| MIPROv2 candidate | Instruction state plus bootstrapped demo candidate selection |
The comparison table should not be filled with fake numbers. A real report should show:
program id/version
optimizer/config
visible examples
holdout examples
aggregate score
per-case failures
artifact fingerprint
model/provider/config
DSPy version
If MIPROv2 wins on dev and loses on holdout, the honest conclusion is that the development win did not survive the independent comparison. Possible explanations include overfitting, metric weakness, dataset size, domain mismatch, or ordinary variation.
With this book’s one-case holdout, we cannot distinguish those explanations reliably. The failure should trigger diagnosis and more evidence, not a confident causal story about why MIPROv2 succeeded or failed.
7. Persist the artifact
Save the candidate and the manifest together:
The artifact must support two different questions:
What Usually Goes Wrong
| Symptom | Likely cause | How to diagnose it | What to change |
|---|---|---|---|
| MIPRO consumes far more calls than expected | auto mode was mistaken for a fixed call budget |
Inspect resolved trials/candidates and LM usage | Record realized search settings; use explicit settings when tighter control is required |
| Useful labeled demos are missing or malformed | Training rows do not match leaf predictor Signatures | Compare train-row fields with predictor inputs/outputs | Use stage-specific labels or keep max_labeled_demos=0 |
| Dev score jumps to a perfect value immediately | Dev set is too small | Count validation cases and inspect score granularity | Treat it as a harness check; expand dev data before trusting search |
| Dev score rises, holdout falls | Search result did not generalize beyond dev | Compare per-case failures by split | Gather more data and inspect overfitting/metric weaknesses before changing the protocol |
| Candidate cannot be reproduced | State was saved without complete optimizer/runtime provenance | Inspect artifact and manifest together | Save candidate-state fingerprint plus data/metric/model/search metadata |
| Unexpected model behavior appears during optimization | prompt_model and task_model identities were conflated |
Record both model/config paths separately | Pin and audit each optimization dependency |
| Instruction candidates encode bad style | Training data or proposed instructions are narrow/noisy | Inspect optimized signature state and source examples | Filter data, constrain the task contract, or revise the metric on development data only |
Conclusion
We gained instruction-level optimization under an explicit search boundary. MIPROv2 can propose instruction state and bootstrapped demonstration combinations, score them on development evidence, and return a candidate whose state and search provenance can be persisted.
We removed the assumption that human-authored instruction text is fixed forever, but we did not remove experimental discipline: development wins are optimization evidence, not promotion evidence.
The remaining limitation is the feedback channel. A scalar score can rank candidate states without explaining what failed or how the next proposal should change. That handoff leads directly to Chapter 13, where optimization can consume richer textual feedback as well as scalar scores.
