Build a Self-Improving Engineering Program
Build a Self-Improving Engineering Program
We can now assemble the book.
The target is not an autonomous runtime that rewrites itself, judges its own work, and deploys the result. That collapses proposal, evidence, and authority into one component.
The stronger meaning is:
A self-improving engineering program participates in a governed evidence loop: production and evaluation create auditable evidence, optimizers propose new program state, independent comparison tests the proposal, and a separate promotion boundary decides what may become active.
Self-improvement is therefore a property of the lifecycle, not permission for the currently active program to modify itself in place.
The capstone task is repository repair.
Input:
repository
issue / requested engineering change
constraints
Output:
diagnosis
evidence
proposed intervention
candidate patch
validation result
confidence / uncertainty
The real success criterion is external:
Did the change resolve the motivating problem
without causing unacceptable regressions?
1. The insufficient baseline
The smallest DSPy version is tempting:
import dspy
repair = dspy.Predict("issue, repository_context -> diagnosis, patch")
It fails for reasons the book has already exposed:
contract too vague
context too large
no tool boundary
no validation loop
no split discipline
no promotion boundary
patch similarity confused with success
So we build the real program from parts.
2. Capstone architecture
CoCoder’s current candidate_patch_generation EngineeringProgram demonstrates the same separation at a larger scale. The program definition is an explicit, versioned contract with objective, input/output schemas, constraints, runtime policy, validation contract, and optimization surface. ProgramRuntime executes one concrete version; ProgramOptimizer is a separate extension point that proposes a candidate version.
The direct runtime delegates candidate generation through the existing provider-neutral model-execution layer. Independent candidate evaluation then reconstructs the source revision in an isolated workspace, materializes the patch, applies scope and validation gates, analyzes the candidate, computes EngineeringDelta, and persists the resulting evidence.
DSPy therefore sits behind the optimization boundary. It is not the EngineeringProgram abstraction, the repository validator, or the promotion authority.
3. Program contracts
The contracts deliberately exclude promotion_decision, historical outcome fields, future repository state, and any gold patch. Those belong to evaluation and governance.
They also avoid treating self-reported confidence as correctness. The program may report uncertainty or missing evidence, but only external validation can establish whether a patch applies, satisfies the concern, and avoids regressions.
4. Validation loop
The validation contract is frozen before optimization. Each candidate attempt executes against a clean reconstruction of the case revision:
5. Optimization dataset
A repository-repair case should separate the state available when the decision was made from evidence learned afterward:
6. Choose the optimizer by failure
Do not stack optimizers for decoration.
| Failure | Reasonable intervention |
|---|---|
| Useful predictor-level traces can be accepted by a reliable metric | BootstrapFewShot for bootstrapped demos |
| Trusted stage-specific labels match predictor Signatures | Labeled few-shot demonstrations may be appropriate |
| Instructions are systematically weak and dev data is adequate | MIPROv2 |
| Scored components receive rich, non-leaking diagnostics | GEPA |
| Evidence selection misses the root cause | Improve retrieval/tools before prompt optimization |
| Validation cannot express the motivating concern | Fix the validation contract before optimizing the program |
For a composed repository-repair program, do not assume one final gold patch is automatically a labeled demo for every internal predictor. Stage labels must match the Signatures they supervise, just as in Chapter 11.
GEPA becomes attractive when validation failures can produce actionable feedback such as:
7. Frozen experiment and promotion
The manifest freezes the experimental contract:
The protocol:
8. A small executable manifest guard
The capstone needs ordinary tests around the experiment boundary.
This is not a substitute for CoCoder’s full experiment service. It demonstrates three minimum invariants: the experiment is identified by frozen configuration, case IDs are disjoint, and source groups are disjoint as well.
The guard still cannot detect semantic leakage hidden inside free text, a future repository snapshot exposed by a tool, or an answer-bearing memory record. Those require the provenance and adversarial-firewall checks from Chapter 16. The manifest guard proves structure; it does not prove innocence.
9. Production evidence loop
The finished loop:
What Usually Goes Wrong
| Symptom | Likely cause | How to diagnose it | What to change |
|---|---|---|---|
| Patch is syntactically valid but concern persists | Validation checks artifact shape, not motivating behavior | Compare the issue contract with executed checks | Add a behavior-focused concern test before optimizing again |
| Revision loop keeps producing slightly different patches | Failure interpretation has no bounded stopping rule | Inspect attempt count, repeated evidence, and validation deltas | Enforce attempt/resource budget and return insufficient evidence |
| Tests pass but scope changed | Scope is not a hard gate | Compare changed files, change-set fingerprint, and engineering delta | Reject before broader promotion comparison |
| Bootstrap/MIPRO demos behave strangely | Final repair rows do not match internal predictor Signatures | Inspect predictor demo fields and source lineage | Use stage-compatible labels or bootstrapped traces |
| GEPA improves unscored stages | Optimizer can mutate components absent from the objective | Compare mutation surface with metric-observed outputs | Freeze those components or add valid metrics before optimization |
| Optimizer score improves but candidate is rejected | Independent governance evidence found a regression or insufficient evidence | Read per-dimension comparison and case failures | Fix the program/evidence; do not tune promotion policy on the same holdout |
| Program uses a historical or future answer | Memory, retrieval, feedback, or tools crossed the decision-time boundary | Trace source revisions, memory fields, feedback lineage, and group fingerprints | Reject the run and rebuild the experiment under the firewall |
| Candidate passes offline but cannot deploy | Deployment bundle does not reconstruct evaluated dependencies | Compare candidate-state and deployment fingerprints | Fix deployment parity before activation |
| Production observations immediately become training labels | Evidence-maturation boundary was skipped | Trace dataset rows back to production runs and timestamps | Reconstruct decision-time views and derive labels through a versioned policy |
| Active behavior cannot be explained | Run is not tied to immutable deployment/promotion lineage | Trace request → deployment → candidate → comparison → promotion | Require fingerprints and durable records at every transition |
Conclusion
We did not build a prompt optimizer. We built a governed engineering lifecycle around a language-model program.
The completed system has a task contract, bounded capabilities, evidence-selection policy, typed program state, validation contract, optimization dataset, metric, optimizer, frozen experiment manifest, candidate artifact, independent holdout evidence, comparison policy, promotion lineage, deployment bundle, production telemetry, feedback-maturation path, and rollback.
No single model call owns that system. The LM handles uncertainty where generative reasoning is useful. Ordinary software owns identity, boundaries, validation, persistence, comparison, and authority.
The final thesis is simple:
A language-model application becomes engineerable when behavior can be specified, evidence can be traced, changes can be proposed without self-approval, improvements can survive independent comparison, and production outcomes can become the audited evidence for the next experiment.
