You Cannot Optimize What You Cannot Measure

Chapter 7 turned examples into data. We now have rows, labels, provenance, and splits.

That does not yet give us evidence. It gives us the material from which evidence can be produced.

The next step is a baseline evaluation:

frozen program
      +
frozen cases
      +
metric
      ↓
per-case results
      ↓
aggregate result
      ↓
failure inspection

No optimization happens in this chapter. Evaluation answers one question:

How well did this frozen program perform under this frozen protocol?

1. The baseline must be named

A baseline is not “whatever code ran yesterday.” For an evaluation claim, we need enough identity to distinguish the program, its LM execution boundary, the dataset split, and the metric.

For the teaching experiment:

2. A first metric for baseline smoke tests

We will build the serious metric in chapter 9. For now we need a conservative smoke-test metric that catches obvious failures without pretending to measure full editorial quality.

This metric does not say the rewrite is good. It says the output was non-empty and passed two narrow lexical checks.

Even those checks have limits. Case-insensitive term matching can detect a renamed Jalen or an explicitly forbidden word, but it does not prove semantic preservation, coreference, or voice. The point of a smoke metric is to catch a small class of obvious failures while making its blind spots explicit.

A weak metric is useful only when its role is narrow and named.

3. Evaluate with DSPy

Current DSPy provides dspy.Evaluate(devset=..., metric=...). The evaluator returns an EvaluationResult with:

import dspy

def evaluate_baseline(program, examples):
    evaluator = dspy.Evaluate(
        devset=examples,
        metric=baseline_smoke_metric,
        display_progress=True,
        display_table=False,
        failure_score=0.0,
    )
    return evaluator(program)

For our experiment, the first baseline run belongs on the development set while we are still designing the metric and harness. The holdout set remains untouched until the program, metric, and comparison protocol are frozen.

4. Aggregate score is not enough

Two programs can both report an aggregate 75.0 under a [0,1] metric and still fail in different ways:

def unpack_evaluation(result) -> list[dict]:
    rows = []
    for example, prediction, score in result.results:
        rows.append(
            {
                "case_id": example.case_id,
                "score": float(score),
                "prediction": {
                    "rewritten_text": getattr(prediction, "rewritten_text", None),
                    "rationale": getattr(prediction, "rationale", None),
                    "risk": getattr(prediction, "risk", None),
                    "confidence": getattr(prediction, "confidence", None),
                },
                "reference": {
                    "reference_rewrite": example.reference_rewrite,
                    "required_entities": list(example.required_entities),
                    "forbidden_terms": list(example.forbidden_terms),
                },
            }
        )
    return rows

Writer’s DSPy runtime records the same kind of run evidence around candidates: provider, model, prompt hash, response hash, evidence-packet hash, confidence, risk, fallback state, and latency. Those fields do not prove candidate quality. They make later evaluation attributable: we can identify what produced the candidate and distinguish real LM output from degraded or fallback behavior.

5. Baselines worth comparing

For the editorial program, useful baselines include:

The no-edit baseline is especially useful because a metric may reward copying the original. That is not automatically a metric bug: preserving the source can be better than a destructive edit.

But the interpretation depends on the task contract. If the dataset contains only cases already selected because an edit is required, a metric that consistently prefers no-op behavior is probably missing the intended improvement signal. If “leave unchanged” is a legitimate task outcome, then no-op should be represented explicitly rather than treated as a universal failure.

6. Failure taxonomy

For sentence improvement, inspect failures under categories like:

semantic drift
entity change
constraint violation
unnecessary edit
poor style improvement
invalid output
LM/provider failure
fallback contamination

Writer’s QC code has concrete reason-code families for hard failures such as helper-text leakage, prompt leakage, TODO leakage, markdown artifacts, entity rename risk, quote imbalance, punctuation break, and candidate scope too large. Those are useful because they name failures that deterministic policy can sometimes catch before an LM judge is involved.

The repository-repair version has the same shape:

patch does not apply
tests still fail
unrelated files changed
validation unavailable
candidate generated by fallback

Evaluation can make these differences visible only if the protocol records them. A scalar metric can still hide why a case failed; per-case outputs, deterministic checks, provider state, and failure categories are what make diagnosis possible.

What Usually Goes Wrong

Conclusion

We designed a baseline evaluation protocol: identify the program and LM boundary, freeze the cases and metric, run the program, retain per-case results, and inspect failures before optimization.

We removed the assumption that optimization should begin before the current program can be measured under a reproducible protocol.

The tiny teaching corpus is enough to exercise that machinery, not to establish performance. The next problem is harder: even a perfectly reproducible evaluation is misleading if the metric rewards the wrong behavior.