An AI agent says:

Done. The task is complete.

That sentence is almost worthless.

The agent may have:

• edited the wrong file,
• changed the right file incorrectly,
• skipped part of the request,
• broken another subsystem,
• failed to save its work,
• misread a tool result,
• passed a stale test,
• inspected the wrong environment,
• or simply decided that its own answer looked convincing.

The central problem is simple:

The system that produced the answer should not be the only system deciding whether the answer is correct.

This is one of the most important ideas in agent engineering.

A useful agent does not merely act.

It acts, observes, and then attempts to prove that the intended outcome actually occurred.

request
   ↓
agent action
   ↓
environment changes
   ↓
verification
   ↓
pass / fail / unknown

The difference between a demo agent and a production agent is often not the sophistication of the planner.

It is the quality of the verification boundary.

This post builds that boundary from first principles.

We will cover:

• why model self-evaluation is weak evidence,
• how to define success before execution,
• deterministic verification,
• behavioral verification,
• invariant checking,
• differential checks,
• multi-layer verification,
• stale evidence,
• partial success,
• false positives and false negatives,
• verification for coding agents,
• browser agents,
• research agents,
• data agents,
• support agents,
• DevOps agents,
• and how to measure whether a verifier is actually useful.

The goal is not to make agents pessimistic.

The goal is to stop confusing confidence with evidence.

The failure mode people actually encounter

A large class of agent bugs has the same shape:

agent performs action
        ↓
agent sees plausible output
        ↓
agent infers success
        ↓
agent stops
        ↓
real task is still broken

Typical searches look like:

• AI agent says task complete but it is not
• coding agent says tests pass but code is broken
• LLM agent hallucinated successful tool execution
• agent claims browser form submitted but nothing happened
• AI agent evaluates its own answer incorrectly
• agent says deployment succeeded but service is down
• agent completed only part of task
• LLM verifier agrees with wrong answer

The common mistake is treating the model’s statement about the world as if it were the world.

It is not.

The model only has observations.

Those observations may be:

• incomplete,
• stale,
• ambiguous,
• summarized incorrectly,
• or interpreted incorrectly.

So we need a stronger architecture.

1. Separate action from evidence

Suppose a coding agent edits a function.

The model might say:

I fixed the bug.

That is a claim.

A test result is evidence.

A diff is evidence.

A type checker result is evidence.

A reproduced failing case that now passes is evidence.

A model’s explanation of why the code should work is not the same category.

This gives us our first rule:

Agent output is a proposal or claim until external evidence supports it.

We can represent this explicitly.

from dataclasses import dataclass
from enum import Enum


class VerificationStatus(str, Enum):
    PASS = "pass"
    FAIL = "fail"
    UNKNOWN = "unknown"


@dataclass
class VerificationResult:
    status: VerificationStatus
    evidence: list[str]
    reason: str

Already this is better than:

success = model_says_done

Now success is an object with evidence.

2. Define success before the agent starts

A verifier cannot verify an undefined goal.

Consider:

Fix the checkout bug.

What counts as success?

Possible interpretations:

• the failing unit test passes,
• all checkout tests pass,
• all repository tests pass,
• the checkout page loads,
• the payment API returns success,
• no new exceptions appear,
• the original reproduction no longer fails.

These are not equivalent.

A useful agent runtime should turn the request into explicit success criteria.

For example:

@dataclass
class SuccessCriterion:
    name: str
    required: bool
    verifier: str


criteria = [
    SuccessCriterion(
        name="original reproduction passes",
        required=True,
        verifier="reproduction_test",
    ),
    SuccessCriterion(
        name="checkout test suite passes",
        required=True,
        verifier="pytest_checkout",
    ),
    SuccessCriterion(
        name="no new type errors",
        required=True,
        verifier="mypy_checkout",
    ),
]

This changes the task from:

make something that looks fixed

into:

produce a state that satisfies these conditions

That is much closer to ordinary software engineering.

3. Completion is a predicate, not a sentence

A useful mental model is:

success = predicate(environment_state)

For a coding task:

success = tests_pass and lint_pass and reproduction_fixed

For a browser task:

success = order_status == "confirmed"

For a data task:

success = row_count_expected and schema_valid and reconciliation_error == 0

For a deployment:

success = deploy_exit_code == 0 and health_check_ok and error_rate_normal

This is much stronger than:

success = agent_says_success

4. The simplest useful verifier

Let us build one.

from dataclasses import dataclass
from enum import Enum
from typing import Callable


class VerificationStatus(str, Enum):
    PASS = "pass"
    FAIL = "fail"
    UNKNOWN = "unknown"


@dataclass
class VerificationResult:
    name: str
    status: VerificationStatus
    evidence: str


Verifier = Callable[[], VerificationResult]


def run_verifiers(verifiers: list[Verifier]) -> list[VerificationResult]:
    return [verify() for verify in verifiers]


def overall_status(results: list[VerificationResult]) -> VerificationStatus:
    if any(r.status == VerificationStatus.FAIL for r in results):
        return VerificationStatus.FAIL

    if any(r.status == VerificationStatus.UNKNOWN for r in results):
        return VerificationStatus.UNKNOWN

    return VerificationStatus.PASS

Notice something important.

UNKNOWN exists.

Production systems often force verification into:

true / false

But sometimes the system genuinely cannot tell.

Examples:

• a third-party API timed out,
• a browser confirmation element never appeared,
• monitoring data has not arrived yet,
• a test environment is unavailable,
• a source cannot be accessed,
• the verifier itself crashed.

Treating UNKNOWN as PASS is dangerous.

Treating UNKNOWN as FAIL may also be wrong.

The runtime should preserve uncertainty.

5. Model self-evaluation is not independent verification

Consider this pattern:

answer = model.generate(task)

verdict = model.generate(
    f"Is this answer correct?\n\n{answer}"
)

This can still be useful.

But it is not independent evidence.

The same model family may share:

• the same blind spots,
• the same assumptions,
• the same missing context,
• the same preference for fluent explanations,
• and the same false belief that produced the answer.

If the generator hallucinates that a function exists, the evaluator may happily reason from the same hallucinated function.

A better evidence hierarchy is often:

real environment signal
        ↓
deterministic verifier
        ↓
independent external source
        ↓
specialized learned evaluator
        ↓
separate LLM judge
        ↓
same-model self-critique

This is not an absolute ranking for every domain.

But it is a useful default.

6. Verification should observe the result, not the explanation

Suppose an agent modifies code.

Weak verification:

Explain why your patch fixes the bug.

Stronger verification:

Run the previously failing test.

Even stronger:

Run the reproduction.
Run adjacent tests.
Run static checks.
Inspect the actual diff.

This principle generalizes.

For a browser agent:

Weak:

Did you submit the form?

Stronger:

Does the server-side order exist?

For a support agent:

Weak:

Did you refund the customer?

Stronger:

Does the payment system show the refund transaction?

For a deployment agent:

Weak:

Did deployment complete?

Stronger:

Is the new version live and healthy?

7. Verify the intended effect, not merely the action

This is subtle.

Suppose a tool call returns:

{
  "success": true,
  "message": "deployment command completed"
}

That verifies the command.

It does not verify the service.

Likewise:

git push succeeded

does not prove:

CI passed

And:

CI passed

does not prove:

production is healthy

We need layers.

action succeeded
       ↓
state changed
       ↓
expected behavior observed
       ↓
user goal satisfied

Each arrow may require a different verifier.

8. Local success vs global success

One of the easiest agent bugs to miss is local success.

Example:

step 1: update file      PASS
step 2: run test         PASS
step 3: commit change    PASS

The agent concludes:

task PASS

But maybe the actual requirement was:

fix bug without changing API behavior

The patch could pass the targeted test while breaking an external contract.

This is the same distinction we saw earlier with planning:

Step completion is not goal completion.

Verification should therefore happen at multiple levels.

step-level verification
        ↓
component-level verification
        ↓
goal-level verification

9. Verification layers

A practical agent can use several layers.

Layer 1: syntactic verification

Does the output parse?

Examples:

• JSON parses,
• Python compiles,
• SQL parses,
• configuration is valid YAML.

Layer 2: structural verification

Does it match the expected shape?

Examples:

• required fields exist,
• schema matches,
• function signature preserved,
• table columns present.

Layer 3: semantic verification

Does the result mean what we need it to mean?

Examples:

• order status is confirmed,
• target test passes,
• research claim is supported by source,
• reconciliation equals expected total.

Layer 4: behavioral verification

Does the system behave correctly when exercised?

Examples:

• integration tests,
• browser flow,
• API request,
• replayed production incident.

Layer 5: invariant verification

Did we preserve constraints that must remain true?

Examples:

• no unauthorized file access,
• account balance preserved,
• latency within bound,
• schema backward compatibility maintained.

Layer 6: regression verification

Did the change fix one thing while breaking something else?

Examples:

• broader test suite,
• old benchmark cases,
• historical tasks,
• golden outputs.

No single verifier gives every guarantee.

10. A verification pipeline

We can represent verification as a sequence of gates.

from dataclasses import dataclass
from typing import Callable


@dataclass
class Gate:
    name: str
    run: Callable[[], VerificationResult]
    required: bool = True


def verify_goal(gates: list[Gate]):
    results = []

    for gate in gates:
        result = gate.run()
        results.append(result)

        if gate.required and result.status == VerificationStatus.FAIL:
            break

    return results

This makes the evidence path visible.

A coding agent might define:

gates = [
    Gate("syntax", verify_syntax),
    Gate("targeted_test", run_targeted_test),
    Gate("type_check", run_type_check),
    Gate("regression_suite", run_regression_tests),
]

A browser agent might define:

gates = [
    Gate("page_loaded", verify_page_state),
    Gate("form_values", verify_form_values),
    Gate("submission", verify_submission),
    Gate("server_record", verify_server_record),
]

The runtime can now explain exactly where verification failed.

11. Don’t let the agent choose the easiest verifier

If the model controls both action and verification strategy, it may accidentally optimize for easy evidence.

Suppose the task is:

fix the API bug

The model could choose:

verify by reading the edited function

instead of:

verify by reproducing the API request

The first is cheaper and more likely to look good.

The second is more relevant.

So verification policy should often live in runtime code.

verification_policy = {
    "bug_fix": [
        "reproduction",
        "targeted_tests",
        "regression_tests",
    ],
    "data_migration": [
        "row_counts",
        "schema_checks",
        "reconciliation",
    ],
}

The model may help instantiate the checks.

But the runtime should control the minimum evidence bar.

12. Verification should be tied to the original request

Agents often drift.

A user asks:

Add pagination without changing the existing response schema.

The agent may successfully add pagination.

But it may also change the schema.

If the verifier only checks:

pagination works

it will declare success.

The success criteria should therefore contain both:

required effect

and:

preservation constraints

For example:

criteria = {
    "must_change": [
        "pagination_supported",
    ],
    "must_preserve": [
        "response_schema",
        "authentication_behavior",
    ],
}

This is especially important for coding agents.

13. Verification for coding agents

Coding agents are one of the clearest applications because software already has many external verifiers.

A useful hierarchy might be:

patch
 ↓
syntax / compile
 ↓
static analysis
 ↓
targeted test
 ↓
reproduction case
 ↓
related regression tests
 ↓
full suite if justified
 ↓
benchmark / performance checks

Common failure:

agent edits code
agent reads code
agent says fix looks correct

Better:

agent edits code
runtime runs reproduction
runtime runs tests
runtime checks diff
runtime reports evidence

A simple verifier:

import subprocess


def run_command(name: str, command: list[str]) -> VerificationResult:
    completed = subprocess.run(
        command,
        capture_output=True,
        text=True,
    )

    if completed.returncode == 0:
        return VerificationResult(
            name=name,
            status=VerificationStatus.PASS,
            evidence=completed.stdout[-2000:],
        )

    return VerificationResult(
        name=name,
        status=VerificationStatus.FAIL,
        evidence=(completed.stdout + completed.stderr)[-2000:],
    )

Then:

result = run_command(
    "targeted_test",
    ["pytest", "tests/test_checkout.py", "-q"],
)

The agent cannot talk its way around the return code.

14. But tests can also lie

External verification is stronger than self-evaluation.

It is not infallible.

Tests may be:

• incomplete,
• stale,
• incorrectly scoped,
• skipped,
• mocked too heavily,
• testing the wrong behavior,
• or accidentally bypassed.

So:

Passing a verifier means passing that verifier, not proving universal correctness.

This is why verification provenance matters.

Store:

• what ran,
• against which state,
• in which environment,
• with what inputs,
• at what time,
• and what output it produced.

15. Stale verification is a real agent bug

Imagine:

1. tests pass
2. agent edits file again
3. agent reports tests passed

Those tests no longer verify the final state.

This is a surprisingly easy bug to build into agent runtimes.

The fix is to bind evidence to state identity.

For code, use things like:

• commit SHA,
• tree hash,
• file hashes,
• patch ID,
• workspace snapshot ID.

For data:

• dataset version,
• table snapshot,
• row checksum,
• partition ID.

For deployments:

• artifact version,
• image digest,
• deployment ID.

For browser workflows:

• transaction ID,
• order ID,
• server-side record version.

Verification should answer:

What exact state did this evidence verify?

16. Bind evidence to state

A simple model:

@dataclass
class Evidence:
    verifier: str
    state_id: str
    status: VerificationStatus
    details: str

Before accepting success:

if evidence.state_id != current_state_id:
    status = VerificationStatus.UNKNOWN

This prevents stale evidence from being silently reused.

17. Verification for browser agents

Browser agents frequently confuse action completion with task completion.

Example:

click submit

is not the same as:

submission succeeded

A good browser verifier may inspect:

DOM state
 ↓
network response
 ↓
server-side state
 ↓
confirmation identifier

For example:

clicked “Place order”
        ↓
HTTP 200 received
        ↓
order confirmation page visible
        ↓
order ID present
        ↓
backend lookup confirms order exists

The backend record is usually stronger evidence than the model saying:

The confirmation page looks correct.

18. Verification for research agents

Research tasks are harder because many claims do not have a deterministic test suite.

Still, we can externalize evidence.

Suppose the agent claims:

Company X acquired Company Y in 2025.

Weak verification:

another LLM says this sounds correct

Stronger:

primary source supports claim

A research verification pipeline may check:

claim
 ↓
source exists
 ↓
source is accessible
 ↓
source text supports claim
 ↓
source date is compatible
 ↓
source authority is acceptable
 ↓
claim phrasing does not exceed evidence

The important design principle is:

Verification should trace the claim back to evidence, not merely score how plausible the prose sounds.

19. Research claims need claim-level verification

An answer containing ten claims should not receive one global score.

Represent claims separately.

@dataclass
class Claim:
    text: str
    source_ids: list[str]


@dataclass
class ClaimVerification:
    claim: Claim
    status: VerificationStatus
    evidence: list[str]

Now the runtime can detect:

8 claims supported
1 claim weakly supported
1 claim unsupported

instead of:

answer score = 8.6 / 10

The first is operationally useful.

20. Verification for data agents

Data agents have many deterministic signals available.

Useful checks include:

• row counts,
• uniqueness,
• null rates,
• schema constraints,
• foreign-key consistency,
• reconciliation totals,
• distribution drift,
• checksum comparisons,
• idempotence.

Suppose an agent transforms financial records.

A strong verification pipeline might be:

input rows = 1,250,000
        ↓
transformation runs
        ↓
output rows = expected rows
        ↓
schema valid
        ↓
required keys unique
        ↓
total debit = total credit
        ↓
reconciliation error = 0

The model does not need to decide whether these invariants matter.

The runtime already knows.

21. Verification for support agents

Support agents often perform actions with business consequences.

Examples:

• refund,
• cancel order,
• update address,
• issue credit,
• escalate case,
• change subscription.

A tool returning success=true may still be insufficient.

A refund task might require:

refund tool accepted
       ↓
refund transaction ID exists
       ↓
amount matches requested amount
       ↓
order state updated
       ↓
case note recorded

This is also where authorization verification matters.

A technically successful action can still be invalid if the agent lacked authority to perform it.

22. Verification for DevOps agents

DevOps is another domain where action success and outcome success differ sharply.

Example:

kubectl rollout restart deployment/api

Return code 0 means the command was accepted.

It does not mean:

API healthy

A better chain:

command accepted
      ↓
new pods scheduled
      ↓
pods ready
      ↓
health endpoint passes
      ↓
error rate normal
      ↓
latency acceptable

For a deployment agent, the verifier may be more important than the planner.

23. Application matrix

The pattern is always similar:

verify the state that matters to the user

not:

verify that the agent performed an action

24. Verification should be adversarial enough to matter

Suppose a coding agent writes a test that simply reproduces its implementation assumptions.

Then it writes code that passes that test.

This can create circular validation.

One solution is to preserve some verification logic outside the agent’s control.

Examples:

• hidden tests,
• pre-existing regression suites,
• externally specified invariants,
• independent fixtures,
• immutable acceptance criteria,
• production telemetry.

The verifier should not always be something the agent can redefine after seeing the result.

25. Don’t let the agent rewrite the success criteria after failure

A dangerous loop looks like:

criterion fails
      ↓
agent decides criterion was unnecessary
      ↓
criterion removed
      ↓
agent declares success

Sometimes requirements genuinely need revision.

But that should be an explicit event.

For example:

@dataclass
class CriterionChange:
    criterion_id: str
    old_value: str
    new_value: str
    reason: str
    authorized_by: str

Changing the definition of success should be visible and auditable.

26. Verification can fail too

Suppose the agent is correct but the verifier is broken.

Examples:

• test fixture corrupt,
• browser selector stale,
• metrics endpoint unavailable,
• database replica lagging,
• evaluator model biased,
• source fetch failed.

That is why we need three statuses:

PASS
FAIL
UNKNOWN

And sometimes richer failure metadata:

@dataclass
class VerificationResult:
    name: str
    status: VerificationStatus
    evidence: str
    verifier_error: str | None = None

A verifier crash is not proof that the task failed.

27. False positives and false negatives

A verifier has its own error profile.

False positive

Verifier says:

PASS

but task is actually wrong.

This is especially dangerous because it terminates the agent incorrectly.

False negative

Verifier says:

FAIL

but task is actually correct.

This can cause:

• unnecessary retries,
• destructive rewrites,
• cost blowups,
• oscillation,
• abandonment of good solutions.

So verifier quality must be measured.

28. Measure the verifier, not just the agent

Useful metrics include:

• verification precision,
• verification recall,
• false-pass rate,
• false-fail rate,
• unknown rate,
• stale-evidence rate,
• average verification latency,
• verification cost,
• verification coverage,
• percentage of claims with evidence,
• percentage of required criteria checked,
• regressions caught after local success.

For example:

agent success rate:        78%
false success declarations: 14%

with verifier:
verified success rate:     75%
false success declarations: 2%

That could be a very worthwhile trade.

29. Verification coverage matters

Suppose a task has five required criteria.

The runtime checks two.

Both pass.

Reporting:

verified

would be misleading.

Better:

2 / 5 required criteria verified
3 / 5 unknown

A simple metric:

def coverage(results, required_names):
    observed = {r.name for r in results}
    return len(observed & set(required_names)) / len(required_names)

Completion can require both:

all required checks pass

and:

coverage == 100%

30. Verification provenance

Every verification result should answer:

• who or what produced it,
• what state it verified,
• which inputs were used,
• when it ran,
• which verifier version ran,
• and what evidence was returned.

For example:

@dataclass
class VerificationRecord:
    verifier_name: str
    verifier_version: str
    state_id: str
    input_hash: str
    status: VerificationStatus
    evidence: str
    timestamp: str

This matters when debugging:

Why did the agent think this was successful?

Without provenance, the answer may be impossible to recover.

31. Verification should influence the control loop

Verification is not merely a final report.

It should alter what the agent does next.

act
 ↓
verify
 ├→ PASS → stop
 ├→ FAIL → diagnose / recover
 └→ UNKNOWN → gather evidence / retry verifier / escalate

This creates a verification-driven agent.

A minimal loop:

for step in range(max_steps):
    action = policy.choose(state)
    observation = executor.execute(action)
    state.update(observation)

    verification = verifier.verify(state)

    if verification.status == VerificationStatus.PASS:
        return "success"

    if verification.status == VerificationStatus.FAIL:
        state.add_failure(verification)
        continue

    state.add_uncertainty(verification)

The key is that the verifier observes the environment, not just the model’s explanation.

32. Diagnosis after failure

A verifier saying FAIL is useful.

A verifier saying why is much more useful.

Compare:

FAIL

with:

FAIL
checkout_total expected 49.99
checkout_total observed 59.99

The second gives the agent a concrete recovery signal.

So verification output should be structured when possible.

@dataclass
class FailureDetail:
    criterion: str
    expected: str
    observed: str
    evidence: str

Then the recovery policy can act on actual mismatches.

33. Verification is not the same as critique

We already built critique-and-revision earlier in this series.

They solve different problems.

Critique asks:

What seems weak about this candidate?

Verification asks:

Did the required condition actually hold?

Critique is often interpretive.

Verification should be evidential.

critique
    ↓
possible defect

verification
    ↓
observed pass/fail condition

Use critique when correctness is hard to measure directly.

Use verification whenever the environment exposes a measurable condition.

34. Verification is not the same as scoring

A scorer might say:

candidate quality = 0.87

A verifier might say:

required API test failed

The scorer helps rank alternatives.

The verifier helps determine whether the task is complete.

In search agents, both can be useful.

partial branch
   ↓
score
   ↓
choose branch
   ↓
complete solution
   ↓
verify

Do not confuse a high score with a passing acceptance test.

35. Verification and Best-of-N

Best-of-N becomes stronger when selection uses external evidence.

Instead of:

generate 5 answers
LLM judge picks one

we can do:

generate 5 candidates
run verifier on each
discard failing candidates
rank survivors

For coding tasks:

5 patches
 ↓
compile
 ↓
targeted tests
 ↓
regression tests
 ↓
rank survivors by simplicity / performance / diff size

Verification converts search from aesthetic selection into evidence-driven selection.

36. Verification and planning

Planning agents should verify both:

step outcomes

and:

final goal

A plan can contain:

@dataclass
class PlanStep:
    id: str
    action: str
    expected_result: str
    verifier: str

After each step:

execute
 ↓
verify expected result
 ↓
mark complete only if evidence supports it

This prevents the planner from treating a merely attempted action as a completed dependency.

37. Verification and memory

Memory should not store unverified success as fact.

Bad episodic memory:

2026-08-08: fixed checkout bug successfully

Better:

2026-08-08:
patch applied
reproduction PASS
checkout tests PASS
full suite UNKNOWN

Now later agents can reason from evidence quality.

The memory system should distinguish:

claimed success

from:

verified success

38. Verification and search

Search becomes much stronger when partial or complete branches can be tested against the environment.

Coding example:

branch A → 8 tests fail
branch B → 3 tests fail
branch C → 0 targeted tests fail

That is a much better search signal than:

LLM thinks branch B sounds elegant

Verification therefore provides value signals for search.

But remember:

0 targeted tests fail

still may not equal full task success.

The final acceptance gate should remain explicit.

39. Verification should sometimes happen before expensive actions

Verification is not only post-condition checking.

We can also verify preconditions.

Before a destructive action:

Is target environment correct?
Is backup available?
Is user authorized?
Is version expected?
Is operation reversible?

For example:

preconditions = [
    verify_target_environment,
    verify_backup_exists,
    verify_current_version,
]

Then execute only if all required checks pass.

This turns verification into a safety boundary.

40. Two-sided verification

A robust action often has:

preconditions
    ↓
execute
    ↓
postconditions

Example: database migration.

pre:
  schema version == 41
  backup exists
  migration lock acquired

execute:
  apply migration 42

post:
  schema version == 42
  row counts consistent
  constraints valid
  application health good

The model can propose the migration.

The runtime owns the gates.

41. Verification budgets

Verification has a cost.

Running an entire test suite after every tiny edit may be wasteful.

A useful strategy is staged verification.

cheap checks first
       ↓
if pass
       ↓
more expensive checks
       ↓
if still pass
       ↓
full acceptance checks

For coding:

syntax
 ↓
targeted test
 ↓
related module tests
 ↓
full suite

For data:

schema
 ↓
sample checks
 ↓
partition reconciliation
 ↓
full reconciliation

This reduces cost while preserving a high final evidence bar.

42. Cheap verifier first

Suppose:

• syntax check costs 0.1 seconds,
• targeted tests cost 3 seconds,
• full suite costs 15 minutes.

Run them in that order.

Do not spend 15 minutes proving a patch fails to parse.

This seems obvious in normal software engineering.

Agent systems should preserve the same discipline.

43. Verification cascades

We can formalize staged verification.

@dataclass
class VerificationStage:
    name: str
    verifiers: list[Verifier]
    stop_on_failure: bool = True


def run_verification_cascade(stages):
    all_results = []

    for stage in stages:
        results = run_verifiers(stage.verifiers)
        all_results.extend(results)

        if stage.stop_on_failure and any(
            r.status == VerificationStatus.FAIL
            for r in results
        ):
            break

    return all_results

This also makes verification cost measurable by stage.

44. What if correctness cannot be directly measured?

Some tasks do not expose clean acceptance tests.

Examples:

• writing quality,
• strategic recommendations,
• architecture design,
• novel research synthesis,
• ambiguous planning.

Then we need weaker forms of evidence.

Possible approaches:

• rubric-based evaluation,
• independent judges,
• adversarial review,
• multiple evaluators,
• consistency checks,
• source support,
• constraint verification,
• human review.

The important thing is to preserve the evidence hierarchy.

Do not pretend a subjective judge is equivalent to a deterministic invariant.

45. Use objective islands inside subjective tasks

Even subjective tasks often contain objectively checkable subproblems.

Architecture review:

Subjective:

Is this architecture good?

Objective islands:

• does referenced component exist?
• does dependency direction match repository?
• do tests cover claimed behavior?
• does benchmark output support latency claim?
• is interface actually public?

Research synthesis:

Subjective:

Is this interpretation persuasive?

Objective islands:

• are citations real?
• do sources support claims?
• are dates correct?
• are quotations accurate?

Use deterministic checks wherever possible, then reserve model judgment for the residue.

46. Human verification is still verification

For some high-impact tasks, the correct verifier is a person.

Examples:

• sending legally consequential messages,
• irreversible infrastructure changes,
• financial approvals,
• medical decisions,
• public publication,
• deleting important data.

The runtime can represent this explicitly:

status = AWAITING_HUMAN_APPROVAL

rather than pretending autonomy is always the goal.

An agent that knows when it lacks sufficient verification is more useful than one that always produces a confident DONE.

47. Don’t confuse autonomy with reliability

A highly autonomous agent may:

• make more decisions,
• use more tools,
• execute more steps,
• require less human interaction.

None of those imply that it is more reliable.

A less autonomous system with strong verification may outperform it operationally.

more autonomy
   ≠
more correctness

This is one of the recurring themes of this entire series.

48. Verification-first architecture

We can now describe a stronger agent architecture.

              request
                 ↓
          success criteria
                 ↓
               plan
                 ↓
          proposed action
                 ↓
          precondition checks
                 ↓
              execute
                 ↓
            observation
                 ↓
          postcondition checks
                 ↓
           goal verification
           /       |        \
        PASS      FAIL     UNKNOWN
         ↓          ↓          ↓
        stop     recover   gather evidence

The LLM participates in this system.

It is not the system of record.

49. A complete minimal verification-driven agent

Here is a compact implementation skeleton.

from dataclasses import dataclass, field
from enum import Enum
from typing import Any, Callable


class Status(str, Enum):
    PASS = "pass"
    FAIL = "fail"
    UNKNOWN = "unknown"


@dataclass
class CheckResult:
    name: str
    status: Status
    evidence: str


@dataclass
class AgentState:
    task: str
    step: int = 0
    observations: list[Any] = field(default_factory=list)
    failures: list[CheckResult] = field(default_factory=list)


class VerificationDrivenAgent:
    def __init__(
        self,
        policy,
        executor,
        verifier,
        max_steps: int = 8,
    ):
        self.policy = policy
        self.executor = executor
        self.verifier = verifier
        self.max_steps = max_steps

    def run(self, task: str):
        state = AgentState(task=task)

        for step in range(self.max_steps):
            state.step = step

            verification = self.verifier.verify_goal(state)

            if verification.status == Status.PASS:
                return {
                    "status": "success",
                    "verification": verification,
                    "state": state,
                }

            action = self.policy.choose_action(
                state=state,
                verification=verification,
            )

            precondition = self.verifier.verify_action(
                state,
                action,
            )

            if precondition.status == Status.FAIL:
                state.failures.append(precondition)
                continue

            observation = self.executor.execute(action)
            state.observations.append(observation)

            postcondition = self.verifier.verify_observation(
                state,
                action,
                observation,
            )

            if postcondition.status == Status.FAIL:
                state.failures.append(postcondition)

        final = self.verifier.verify_goal(state)

        return {
            "status": "success"
            if final.status == Status.PASS
            else "incomplete",
            "verification": final,
            "state": state,
        }

The implementation is deliberately ordinary.

The interesting part is not the class hierarchy.

It is the contract:

The agent is not allowed to declare success without the verifier.

50. The verifier can be domain-specific

A generic interface is enough.

class Verifier:
    def verify_goal(self, state):
        ...

    def verify_action(self, state, action):
        ...

    def verify_observation(self, state, action, observation):
        ...

Then implementations differ by software type.

CodingVerifier
BrowserVerifier
ResearchVerifier
DataVerifier
SupportVerifier
DeploymentVerifier

The agent loop stays the same.

The evidence changes.

This mirrors an earlier lesson from loop control:

The control structure is generic; the progress and verification signals are domain-specific.

51. Coding verifier example

class CodingVerifier:
    def verify_goal(self, state):
        tests = run_tests()

        if tests.returncode != 0:
            return CheckResult(
                name="goal",
                status=Status.FAIL,
                evidence=tests.stderr,
            )

        reproduction = run_reproduction()

        if not reproduction.fixed:
            return CheckResult(
                name="goal",
                status=Status.FAIL,
                evidence=reproduction.details,
            )

        return CheckResult(
            name="goal",
            status=Status.PASS,
            evidence="targeted tests and reproduction pass",
        )

In a real system, you would probably also bind this to a workspace hash.

52. Research verifier example

class ResearchVerifier:
    def verify_claim(self, claim, sources):
        if not sources:
            return CheckResult(
                name="claim_support",
                status=Status.FAIL,
                evidence="no supporting source",
            )

        support = find_supporting_passages(claim, sources)

        if not support:
            return CheckResult(
                name="claim_support",
                status=Status.UNKNOWN,
                evidence="sources retrieved but no clear support found",
            )

        return CheckResult(
            name="claim_support",
            status=Status.PASS,
            evidence=str(support),
        )

Again, the important thing is that the evidence comes from the source set rather than the model’s confidence.

53. Verification should produce inspectable traces

A production trace might look like:

{
  "task": "Fix checkout total bug",
  "state_id": "tree:91ae...",
  "checks": [
    {
      "name": "syntax",
      "status": "pass"
    },
    {
      "name": "reproduction",
      "status": "pass"
    },
    {
      "name": "checkout_tests",
      "status": "pass"
    },
    {
      "name": "full_suite",
      "status": "unknown",
      "reason": "suite exceeded budget"
    }
  ]
}

Then the final status might be:

PARTIALLY VERIFIED

rather than falsely claiming universal success.

54. Named completion states are better than done=True

Useful states include:

VERIFIED_SUCCESS
VERIFIED_FAILURE
PARTIAL_SUCCESS
UNKNOWN
BUDGET_EXHAUSTED
VERIFIER_ERROR
AWAITING_APPROVAL
ENVIRONMENT_UNAVAILABLE

This gives downstream systems a much clearer contract.

55. Verification and retries

If verification fails, do not blindly retry the original action.

The failure should become evidence for recovery.

Example:

verification failure:
expected status code 200
observed status code 500

The next action should probably be diagnostic.

inspect logs

not:

repeat deployment

This connects verification directly to the earlier loop-control lesson.

56. Verification-driven recovery

A useful sequence is:

FAIL
 ↓
classify failure
 ↓
choose recovery strategy
 ↓
act
 ↓
verify again

Failure classes might include:

implementation defect
environment defect
missing prerequisite
stale state
permission problem
verification failure
unknown

Now recovery becomes targeted rather than repetitive.

57. Avoid verification loops

Verification itself can loop.

Example:

verify
 ↓
unknown
 ↓
retry verifier
 ↓
unknown
 ↓
retry verifier

So verifier retries need budgets too.

max_verification_attempts = 3

After that:

UNKNOWN

should remain unknown.

Do not manufacture certainty through repeated sampling.

58. A verifier should sometimes disagree with the agent

If your verifier almost never disagrees with the agent, ask why.

Possibilities:

• the agent is exceptionally good,
• the verifier is too weak,
• the verifier sees the same evidence and makes the same assumptions,
• the verification criteria are trivial,
• or the verifier is effectively rubber-stamping.

Measure disagreement.

agent claims success: 100 tasks
verifier agrees:       94
verifier fails:         4
verifier unknown:       2

Those six disagreements are extremely valuable debugging cases.

59. Verification disagreements are training data

Every mismatch creates evidence.

agent: success
verifier: fail

This is a high-value case for:

• prompt improvement,
• policy training,
• tool design,
• scorer calibration,
• memory updates,
• failure taxonomy,
• benchmark expansion.

Verification therefore improves not only runtime reliability but also the learning loop around the agent.

60. Benchmark verification separately

Suppose you improve the agent and the verifier simultaneously.

You may no longer know why results improved.

Keep benchmark cases where ground truth is independently known.

Then test:

agent without verifier
agent + verifier A
agent + verifier B

And also:

verifier A against known outcomes
verifier B against known outcomes

This prevents verifier quality from becoming invisible.

61. Controlled experiment

For a coding agent, compare:

A: one-shot patch + self-report
B: one-shot patch + targeted test
C: agent loop + targeted test
D: agent loop + targeted + regression verification

Measure:

• verified success,
• false success declarations,
• false failures,
• average model calls,
• average verifier calls,
• latency,
• cost,
• regression rate,
• recovery success after failed verification.

A sophisticated loop that still produces many false success declarations is not production-ready.

62. Failure injection

Verification systems should be tested against deliberately bad outcomes.

Inject:

• stale test results,
• partial writes,
• malformed outputs,
• successful command with failed service,
• wrong environment,
• missing source,
• contradictory source,
• failed backend update despite UI success,
• schema-valid but semantically wrong data,
• correct local result with broken regression.

Then ask:

Did the verifier catch it?

This is much stronger than only testing successful cases.

63. Evidence before confidence

Agents often produce confidence-like language:

This should work.

I'm confident the issue is resolved.

Everything looks correct.

Those statements can be useful conversationally.

They should not control system state.

System state should come from evidence.

confidence
    ≠
verification

64. The user’s goal is the highest-level verifier

A system can satisfy many internal metrics and still fail the user.

Example:

all tests pass

but the user asked:

make this faster

If latency did not improve, the goal failed.

Or:

all content claims cited

but the user asked:

write a concise executive summary

If the result is 5,000 words, technical correctness alone is insufficient.

Success criteria must include the actual requested outcome.

65. What should be deterministic?

A useful rule:

Make verification deterministic whenever the domain permits it.

Good deterministic checks:

• exact schema,
• unit tests,
• checksums,
• counts,
• permissions,
• file existence,
• command exit codes,
• HTTP status,
• API state,
• database invariants,
• version identity.

Use learned or model-based verification for things deterministic checks cannot capture.

Do not invert that order.

66. What should still use an LLM verifier?

LLM evaluators are useful for:

• semantic coverage,
• nuanced instruction following,
• open-ended critique,
• style alignment,
• ambiguous classification,
• comparing multiple acceptable solutions.

But even then, improve independence where possible.

For example:

generator model
      ↓
independent rubric judge
      ↓
source / invariant checks

rather than:

generator asks itself whether it is correct

67. Multiple verifiers can disagree

Suppose:

unit tests: PASS
integration test: FAIL
LLM judge: PASS

Do not average these into:

score = 0.67

They represent different evidence.

Required failures should usually dominate.

if integration_test.status == FAIL:
    overall = FAIL

Verification composition should reflect semantics, not just arithmetic.

68. Required vs advisory checks

Some checks are gates.

Others are signals.

@dataclass
class CheckSpec:
    name: str
    required: bool

Example:

required:
  targeted tests
  API contract test

advisory:
  style score
  complexity score
  benchmark estimate

A poor style score should not override a hard correctness pass unless style is part of the user requirement.

69. Verification quality is task-dependent

There is no universal verifier.

The right verifier depends on:

• the task,
• the environment,
• the cost of false success,
• the cost of false failure,
• the available evidence,
• the reversibility of actions,
• and the latency budget.

This is why agent architecture should be built around domain contracts, not generic “agent intelligence.”

70. Where verification adds the most value

Verification is especially valuable when:

• actions have side effects,
• mistakes are expensive,
• tasks are multi-step,
• the model cannot directly observe the final state,
• environments are noisy,
• tools may partially fail,
• user requirements contain preservation constraints,
• or the agent frequently declares success prematurely.

It may add less value when:

• the task is purely generative,
• the user will immediately inspect the result,
• failure is cheap,
• or there is no meaningful external signal.

Again:

Do not add a verifier because “agents need verifiers.” Add one where external evidence can materially reduce false success.

71. Do you actually need a verification agent?

Often you do not need a separate agent.

You need ordinary code.

Can correctness be checked deterministically?
        ↓
       yes
        ↓
use deterministic verifier

Only if the residual question remains semantic or ambiguous should you escalate to a learned evaluator or LLM judge.

The strongest verifier may be a ten-line Python function.

72. Verification agent vs verification runtime

There is an important distinction.

A verification runtime can be deterministic infrastructure.

A verification agent can reason about what evidence to gather when fixed checks are insufficient.

Example:

runtime:
  run tests
  check schema
  inspect status codes

verification agent:
  determine which additional evidence is needed
  when current evidence is incomplete

The agent should supplement the runtime, not replace deterministic guarantees.

73. Verification can be adaptive

Sometimes the first check fails and reveals which check to run next.

Example:

health check FAIL
    ↓
inspect pod status
    ↓
pods restarting
    ↓
inspect logs
    ↓
configuration error found

That is a good use of agent reasoning.

But the checks themselves remain external observations.

74. Verification-first debugging checklist

If your agent says it succeeded when it did not, inspect these in order.

1. What exact condition defines success?

If you cannot write it down, verification will be vague.

2. Who decides success?

If the answer is “the same model that produced the result,” independence is weak.

3. What external evidence is available?

Tests, API state, database state, metrics, sources, invariants, files, transactions.

4. Is the evidence tied to the final state?

Check for stale results.

5. Are all required criteria covered?

Partial coverage should not become full success.

6. Can the agent redefine the criteria?

If yes, success can drift.

7. What happens on UNKNOWN?

Do not silently convert it to pass.

8. What happens after verification failure?

Use the failure as diagnostic evidence, not merely as a retry trigger.

9. How often is the verifier wrong?

Measure false passes and false failures.

10. Is verification itself too expensive?

Use staged gates and cheap checks first.

75. The larger architecture we have built

At the beginning of this series, an “agent” could be little more than:

prompt
 ↓
model
 ↓
action

We progressively added the machinery needed for reliable software.

structured actions
      ↓
Best-of-N
      ↓
critique + revision
      ↓
planning
      ↓
loop control
      ↓
tool routing
      ↓
memory
      ↓
search
      ↓
verification

But notice what happened.

The system did not become useful because the model became more magical.

It became useful because we added ordinary engineering around uncertainty.

76. The first-principles agent stack

We can now summarize the architecture as:

USER GOAL
   ↓
SUCCESS CRITERIA
   ↓
STATE
   ↓
MODEL / POLICY
   ↓
STRUCTURED ACTION
   ↓
VALIDATION
   ↓
EXECUTION
   ↓
OBSERVATION
   ↓
MEMORY / SEARCH / PLANNING
   ↓
VERIFICATION
   ↓
CONTINUE / RECOVER / STOP

That is an agent system.

The model is important.

But the model is only one component.

77. Final rule

The central lesson of this post is simple:

Do not ask the model whether reality changed. Check reality.

For coding agents, run the tests.

For browser agents, inspect the resulting state.

For research agents, inspect the source evidence.

For data agents, check invariants and reconciliation.

For support agents, verify the transaction.

For DevOps agents, inspect health and telemetry.

And when you cannot know:

UNKNOWN

is a better answer than invented success.

78. Where the series goes next

This post completes the core Agents From First Principles progression.

We now understand the basic mechanisms needed to build an agent that can:

act
plan
revise
route
remember
search
recover
verify

The next step is not simply “more agent.”

The next step is to study more sophisticated ways of allocating computation and coordinating expertise.

That means a new series:

Advanced Agents From First Principles

There we can explore:

• chain-of-thought as computation,
• self-consistency,
• Tree of Thoughts,
• beam search in greater depth,
• Monte Carlo Tree Search,
• evolutionary search,
• mixture of experts at the agent level,
• planner / executor / critic systems,
• multi-agent debate,
• adaptive routing,
• learned agent policies,
• and agents that improve from previous runs.

But the evidence rule will stay the same.

Every added mechanism must solve a measurable failure well enough to justify its cost.