Let the Program Reflect

Chapter 12 ended with a hard limitation. MIPROv2 can search instructions and demonstrations, but the metric we built mostly returns a number.

candidate
   ↓
metric
   ↓
0.42

That number can select among candidates. It cannot explain how the program should change.

The next mechanism is feedback-shaped evaluation:

prediction
    ↓
metric
    ↓
score + diagnostic feedback
    ↓
reflection
    ↓
instruction mutation
    ↓
candidate

Current DSPy describes GEPA as a reflection-driven instruction optimizer. It maintains a population of candidate instruction states, evaluates them on validation examples, and uses textual feedback plus execution traces to propose new instructions for selected predictors.

Current GEPA uses a Pareto-style frontier to decide which candidates are interesting to mutate during search, then returns the candidate with the best aggregate validation score. The search machinery is richer than the core engineering idea:

A failure should carry enough explanation to suggest a program change.

One boundary is important from the start: GEPA is changing predictor instructions. It does not turn the whole Python program into free-form mutable code, and it does not make the reflection model the evaluator.

1. Score and feedback are different

The Chapter 9 metric returned a scalar:

def editorial_metric(example, pred, trace=None) -> float:
    hard_score, failures = hard_constraint_score(example, pred)
    if hard_score == 0.0:
        return 0.0
    ...

For reflective optimization, the metric needs to explain failure:

The helper deliberately reuses editorial_metric(...) for the score. Reflection should add an explanation channel without silently changing the scalar objective from Chapters 9-12.

Current GEPA can call an optimizer-facing metric with:

2. Reflection is not evaluation

There are two roles:

task model
    ↓
produces program behavior
    ↓
metric
    ↓
evaluates behavior

reflection model
    ↓
reads failure evidence
    ↓
proposes program changes

The reflection model can be stronger, slower, or differently configured from the LM used to execute the task program.

Current GEPA accepts reflection_lm=None; with the default proposer it can use the globally configured DSPy LM. A custom instruction proposer can also manage its own reflection mechanism.

That is an API convenience, not a reproducibility recommendation. For an experiment, provide the reflection LM explicitly and record its provider, model, configuration, and role separately from the task-program LM.

The reflection model does not decide what is true. It proposes an edit. The metric and evaluation protocol still decide whether the new candidate is better.

failure
   ↓
reflection proposes change
   ↓
candidate
   ↓
evaluation
   ↓
keep / reject

This is the same governance lesson as CoCoder’s optimizer boundary. The optimizer proposes. Evaluation selects. Promotion is later.

3. A GEPA compile shape

Current GEPA accepts a feedback-shaped metric and one budget form: auto, max_full_evals, or max_metric_calls. Those knobs are alternative units of the same search budget; do not set more than one.

reflection_lm may be provided explicitly or resolved by the configured/default proposal path. track_stats=True attaches detailed_results to the compiled program, exposing candidate lineage and validation scores for audit.

The default component selector is round-robin across predictors. That matters for our composed program because the current editorial metric scores the final rewrite, not the assessor’s risk or confidence. Optimizing a component the metric cannot observe wastes search budget and can produce misleading mutations.

Several choices in this teaching run are deliberate.

component_selector=editing_component_selector restricts reflection to the analyze and rewrite predictors because both can affect rewritten_text, which the current metric scores. The assessor remains fixed until we have labels or metrics for risk, confidence, and risk_reason.

use_merge=False removes GEPA’s candidate-merging mechanism so this chapter isolates reflective instruction mutation. A later experiment could enable merging as a separately declared search mechanism.

auto="light" is still a real search, not proof of quality. GEPA derives a metric-call budget from the program and validation-set size. Our teaching corpus has only one dev case, so a high validation score can still be the result of fitting one example.

This chapter therefore specifies the reflective experiment but reports no performance result.

4. Inspect reflective evidence

Do not treat a GEPA candidate as a black box. With track_stats=True, current GEPA attaches a detailed_results object to the compiled program:

The audit questions should follow the reflective loop:

Writer’s contextual preference judge already demonstrates the surrounding discipline: structured decision fields, deterministic hard policies, prompt fingerprints, model identity, input fingerprints, and raw output capture.

GEPA adds another provenance layer:

What Usually Goes Wrong

Conclusion

We gained a richer optimization feedback channel. The scalar objective remains frozen, while diagnostic feedback and predictor traces give GEPA material for proposing instruction changes.

We removed the assumption that a score alone is enough to guide search, but we did not confuse reflection with evidence of success. GEPA proposes and explores instruction states; independent evaluation still determines whether the selected candidate survived the experiment.

The program still operates mostly on information passed directly into it. Real engineering programs also need bounded ways to observe and act on an environment. That leads to tools and agents.