Example

Measure the gap between what you generated and what you chose

Run the chapter's selection-gap instrument and watch oracle@N rise while the selected success rate stops rising, and the gap between them become the diagnosis.

Generating more candidates is only half of the work, and the chapter’s claim is that the two halves have to be measured separately. oracle@N asks whether a successful candidate was ever generated. selected@N asks whether the selector picked it. The difference is the selection gap, and it is the diagnosis: a large gap means the answers were already there and the selector is losing them.

Run it

The instrument is the chapter’s own code. Save the first block as selection_gap.py, save the second as demo.py beside it, and run the driver with Python 3.11 or later. The standard library only: no model, no network.

Expected outcome

N   oracle@N  selected@N  gap@N   | published: oracle selected gap
1      0.422       0.422  0.000   | 0.61 0.61 0.00
2      0.723       0.665  0.057   | 0.72 0.68 0.04
4      0.925       0.825  0.100   | 0.83 0.74 0.09
8      0.988       0.882  0.105   | 0.89 0.73 0.16

a run with an unverified label is refused, not counted:
  ValueError: verified labels required for eligible candidates: [0]

The shape is the chapter’s: oracle@N climbs with N, the selected rate climbs more slowly, and the gap opens. The absolute numbers are this run’s own, because the run is a seeded simulation with a generator that succeeds 45% of the time and a selector whose score is the truth plus noise. The published figures are in the right-hand columns and are the chapter’s measurement, not this program’s.

The last two lines are the other half of the instrument: a metric that cannot be computed honestly refuses instead of guessing.

The instrument

The chapter’s code, in full. The candidate and run records, and the three functions that make the gap computable.

"""Driver for the agents-from-first-principles chapter 3 selection-gap instrument."""

from dataclasses import dataclass
from statistics import mean
import random


@dataclass
class Candidate:
    value: str
    score: float | None = None
    eligible: bool = True
    verified_success: bool | None = None


@dataclass
class RunRecord:
    task_id: str
    candidates: list[Candidate]
    winner_index: int

    def require_verified_labels(self) -> None:
        unknown = [
            i
            for i, candidate in enumerate(self.candidates)
            if candidate.eligible and candidate.verified_success is None
        ]
        if unknown:
            raise ValueError(
                f"verified labels required for eligible candidates: {unknown}"
            )

    @property
    def oracle_success(self) -> bool:
        """Did the generator produce an eligible successful candidate?"""
        self.require_verified_labels()
        return any(
            c.eligible and c.verified_success is True
            for c in self.candidates
        )

    @property
    def selected_success(self) -> bool:
        """Did the selector return an eligible successful candidate?"""
        winner = self.candidates[self.winner_index]

        if not winner.eligible:
            return False
        if winner.verified_success is None:
            raise ValueError("selected candidate has not been externally verified")

        return winner.verified_success


def oracle_at_n(runs: list[RunRecord]) -> float:
    return mean(r.oracle_success for r in runs)


def selected_at_n(runs: list[RunRecord]) -> float:
    return mean(r.selected_success for r in runs)


def selection_gap(runs: list[RunRecord]) -> float:
    return oracle_at_n(runs) - selected_at_n(runs)

The driver

The driver is not part of the chapter. It simulates a generator and a noisy selector, computes the three figures at four values of N, and prints them beside the chapter’s published table.

"""Driver for the agents-from-first-principles chapter 3 selection-gap instrument."""

from selection_gap import (
    Candidate,
    RunRecord,
    oracle_at_n,
    selected_at_n,
    selection_gap,
)
import random


# The chapter's published table, kept for comparison.
PUBLISHED = {
    1: (0.61, 0.61, 0.00),
    2: (0.72, 0.68, 0.04),
    4: (0.83, 0.74, 0.09),
    8: (0.89, 0.73, 0.16),
}


def simulate(n: int, runs: int = 400, seed: int = 20260923) -> list[RunRecord]:
    """A generator that sometimes succeeds, and a selector that scores noisily.

    The generator succeeds independently per candidate with probability 0.45.
    The score is the truth plus noise, so the selector picks the best-scoring
    candidate, which is the successful one only when the noise does not
    overtake it.
    """
    rng = random.Random(seed + n)
    records = []
    for index in range(runs):
        candidates = []
        for position in range(n):
            success = rng.random() < 0.45
            score = (1.0 if success else 0.0) + rng.gauss(0.0, 0.6)
            candidates.append(
                Candidate(
                    value=f"candidate-{position}",
                    score=score,
                    eligible=True,
                    verified_success=success,
                )
            )
        winner = max(range(n), key=lambda i: candidates[i].score)
        records.append(RunRecord(task_id=f"t{index}", candidates=candidates, winner_index=winner))
    return records


print("N   oracle@N  selected@N  gap@N   | published: oracle selected gap")
for n in (1, 2, 4, 8):
    runs = simulate(n)
    oracle = oracle_at_n(runs)
    selected = selected_at_n(runs)
    gap = selection_gap(runs)
    p_oracle, p_selected, p_gap = PUBLISHED[n]
    print(
        f"{n:<3} {oracle:8.3f} {selected:11.3f} {gap:6.3f}   | "
        f"{p_oracle:.2f} {p_selected:.2f} {p_gap:.2f}"
    )

print()
print("a run with an unverified label is refused, not counted:")
unlabelled = RunRecord(
    task_id="t0",
    candidates=[Candidate(value="c0", score=1.0, eligible=True, verified_success=None)],
    winner_index=0,
)
try:
    unlabelled.oracle_success
except ValueError as error:
    print(f"  ValueError: {error}")

Mechanism and limitations

Documented: the candidate and run records, the three functions, and the published table at N of 1, 2, 4 and 8 are in the chapter’s own text, and the code above is that text’s code. The chapter’s reading of the table is that improvement in oracle@N alone is not improvement, and that the gap is unambiguous: at N of 8 the generator produced a successful option 89% of the time while the selector failed to capture sixteen percentage points of it.

Observed: the run above. The gap opens as N grows, and the instrument refuses a run whose labels are missing rather than scoring it as a failure.

Not shown. The simulation is not the measurement. The generator’s success rate, the noise on the score and the run count are chosen to make the shape visible, and the resulting figures are the simulation’s own, which is why they are printed next to the chapter’s rather than instead of them. A real generator’s candidates are not independent, and a real selector’s errors are not symmetric noise. oracle@N and selection_gap@N need externally verified labels, which the chapter says may require isolated execution or a preview, and this program supplies them by construction.

Read the chapter .

Understand this example

Copy this prompt into your AI tool. No code runs here.

Apply this example

Copy this prompt into your AI tool. No code runs here.