Self-Improving Engineering Program

The problem

Prompt and program optimization make language-model systems easier to improve, but they also make the evidence problem sharper.

A candidate can look better for several different reasons:

• the optimizer found a real improvement;
• the metric rewarded an accidental shortcut;
• the development cases leaked into the candidate;
• a tool exposed information that should have stayed outside the experiment;
• a patch passed visible tests while failing unseen behavior;
• a promotion step quietly became automatic activation.

Those are not the same claim.

The important question is therefore not “did the model make a change?” It is:

What changed, what evidence was the system allowed to see,
which evidence stayed independent, and what justified promotion?

Why the obvious approach fails

The tempting implementation is to put a dspy.Predict or agent around a repository, ask for a patch, run tests and accept the result when the score improves.

That is too little structure for a self-improving engineering system.

The model should not be allowed to certify its own repair. The development tests should not become an endlessly inspected hidden set. A patch artifact should not be the same object as a deployment decision. Tool access should not imply permission to mutate the workspace, inspect future revisions or read evaluator state.

DSPy gives the probabilistic part of the program a clean contract and optimization surface. The surrounding system still has to provide the experimental firewall.

The solution

The Self-Improving Engineering Program is the capstone system from DSPy From First Principles.

It treats repository repair as an evidence loop:

engineering request
    -> frozen fixture repository
    -> bounded read-only inspection
    -> LM diagnosis
    -> candidate patch proposal
    -> isolated patch application
    -> syntax and scope checks
    -> public development validation
    -> independent hidden evaluation when eligible
    -> comparison with the active baseline
    -> promotion decision
    -> no automatic activation

The model does useful uncertain work: diagnosis, localization and repair proposal.

Ordinary software does the parts that need hard boundaries: tool policy, patch application, validation, hidden-evidence isolation, artifact fingerprinting and promotion records.

The core rule is:

the optimizer proposes
independent evaluation establishes evidence
promotion is a separate decision
activation is a separate operation

How it works

The capstone fixture contains a concrete repository defect: invoice_total ignores item quantity.

The repair loop exposes only three read-only tools to the LM:

• repository search;
• bounded file reads;
• symbol inspection.

The LM has no shell, mutation, network or hidden-validation capability. It can inspect the allowed repository evidence, diagnose the issue and propose a replacement for the target file. The system applies that candidate in an isolated workspace, then runs deterministic gates before any independent evaluation is allowed.

The measured run used:

run id: repair-run-20260901-013940
model: ollama_chat/qwen3:latest
DSPy version: 3.3.1
fixture fingerprint: 85b9f606...a2ff8f
candidate artifact fingerprint: 2840b5d5...3b4c85
promotion decision fingerprint: 60df345c...6eedc5
experiment fingerprint: 3d4483a6...242884

Evidence and validation

The capstone is designed so that a failed model repair would still teach something. The success claim is not built into the prompt; it is produced by the evidence loop.

The recorded run produced this outcome:

That result is intentionally narrow. It does not prove that every repository-repair agent will improve itself safely. It proves that this book’s machinery can compose into a measurable repair loop where the LM proposes a candidate and an independent evidence pipeline decides what that candidate means.

Capstone and implementation

The standalone implementation is Evidence Repair:

https://github.com/ernanhughes/evidence-repair

It packages the capstone as a Python project with:

src/evidence_repair/fixtures.py       invoice repair fixture
src/evidence_repair/tools.py          read-only repository tools
src/evidence_repair/dspy_programs.py  DSPy signatures and planner
src/evidence_repair/validation.py     syntax, scope, public and hidden checks
src/evidence_repair/promotion.py      comparison and promotion policy
src/evidence_repair/pipeline.py       run orchestration and artifact writing

The original book reference remains in the book repository:

content/books/dspy-from-first-principles/20-chapter.md
experiments/dspy-from-first-principles/CAPSTONE.md
experiments/dspy-from-first-principles/ch18_repository_repair/
experiments/dspy-from-first-principles/reports/brief-00/report.md

The generated run artifacts are written under .artifacts/dspy-book/ch18/ when the experiment is executed locally. That directory is build output, not the public source of record, so the tracked capstone document summarizes the measured run and points to the executable substrate.

Use it

From the repository root:

python -m pip install -e ".[dev]"
evidence-repair run --out .artifacts\evidence-repair\demo --overwrite

The command creates the fixture repository, runs baseline validation, applies the controlled repair loop and writes evidence records for the candidate, independent evaluation and promotion decision.

For the DSPy path:

python -m pip install -e ".[dev,dspy]"
evidence-repair run --planner dspy --model ollama_chat/qwen3:latest --api-base http://127.0.0.1:11434 --out .artifacts\evidence-repair\dspy --overwrite

What comes next

The next useful step is growing the standalone project surface while preserving the experiment boundary that makes the result meaningful.

Good expansion points are:

• a small CLI that runs the repair loop from a named repository fixture;
• a public artifact bundle containing selected non-secret evidence records;
• more repair fixtures with different failure classes;
• explicit promotion registries for active and candidate programs;
• regression dashboards that separate public development evidence from untouched evaluation;
• comparison policies that reject improvements with hard regressions even when aggregate score rises.

The destination is not autonomous self-modification. It is an engineering program that can search for better behavior while keeping its improvement claims inspectable.