An agent platform can be observable.

It can have replay.

It can have provenance.

It can have incident forensics.

It can have canaries, rollback, circuit breakers, quotas, and distributed coordination.

And the team can still have no disciplined answer to a very basic production question:

How reliable does this agent actually need to be?

Without an answer, every failure feels equally urgent.

Every improvement request competes with every reliability fix.

One team member wants better planning.

Another wants a stronger model.

Another wants more search.

Another wants to fix a verifier false positive.

Another wants to reduce cost.

Another wants to cut p95 latency.

All of them may be right.

But unless the platform has explicit reliability targets, prioritization becomes taste.

That is where Service Level Objectives become useful.

For ordinary services, an SLO might say:

• 99.9% availability,
• p95 latency below 300 ms,
• fewer than 0.1% failed requests.

For agents, availability alone is not enough.

An agent can return HTTP 200 and still be wrong.

It can complete quickly and still use stale evidence.

It can return a plausible answer that the verifier incorrectly accepts.

It can refuse too often.

It can spend ten times the expected budget while still technically succeeding.

Agent reliability therefore needs a richer definition.

The central rule for this post is:

Define reliability around externally verified outcomes, not whether the orchestration completed without throwing an exception.

That gives us a foundation for everything else.

The Search Problem: “AI Agent SLOs and Reliability Metrics”

A naive reliability dashboard for an agent often looks like this:

requests completed: 99.7%
API availability:   99.95%
median latency:     8.2 s
error rate:         0.3%

That can look excellent.

But imagine the real outcomes are:

verified correct:     86%
false success:          4%
UNKNOWN:                7%
hard failure:           3%

The system is operationally healthy and behaviorally weak.

That is why agent SLOs need to begin with outcome semantics.

A useful top-level outcome model is:

PASS
FAIL
UNKNOWN

where:

• PASS means the acceptance criteria were externally verified,
• FAIL means the task was attempted but did not satisfy the acceptance criteria,
• UNKNOWN means the system could not establish enough evidence to safely claim PASS or FAIL.

This is already more useful than a single “success rate.”

1. Start With User-Visible Reliability

The first SLO should describe what the user actually experiences.

Not:

router accuracy >= 94%

Not:

critic agreement >= 90%

Not:

tool success >= 99%

Those may be useful internal metrics.

But they are not the user-facing outcome.

A coding agent might have a top-level SLO such as:

For repository repair tasks in the supported task class:

verified_success_rate >= 95%
false_success_rate    <= 0.5%
p95_latency           <= 180 seconds
p95_cost              <= $1.20

A research agent might instead define:

citation_supported_answer_rate >= 97%
unsupported_claim_rate         <= 0.25%
UNKNOWN_rate                   <= 8%
p95_latency                    <= 90 seconds

A browser agent performing side effects may need a much stricter false-success target:

verified_transaction_success >= 99.5%
duplicate_side_effect_rate   <= 0.01%
ambiguous_completion_rate    <= 0.2%

The correct SLO depends on the task contract.

There is no universal number.

2. False Success Deserves Its Own Budget

This is one of the most important differences between agent reliability and ordinary request reliability.

A normal failed request is usually visible.

A false success is dangerous because the system claims success when the task is actually wrong.

That means:

false_success

should not be hidden inside generic failure rate.

Suppose two systems both have 94% verified success.

System A:

PASS      94%
FAIL       5%
UNKNOWN    1%
false PASS 0.1%

System B:

PASS      94%
FAIL       1%
UNKNOWN    1%
false PASS 4%

Those are not remotely equivalent.

System B is much more dangerous.

The reliability policy should therefore define a separate ceiling:

false_success_rate <= tolerance

For high-risk side effects, that tolerance may need to be extremely small.

3. UNKNOWN Is Not Automatically Failure

One of the easiest mistakes is to punish UNKNOWN exactly like FAIL.

That creates bad incentives.

If an agent knows that saying UNKNOWN hurts its success metric as much as a wrong answer, the optimization pressure encourages unjustified confidence.

That is precisely what we do not want.

Consider:

Agent A
PASS:    90%
FAIL:     5%
UNKNOWN:  5%
false success: 0.2%

Agent B
PASS:    94%
FAIL:     4%
UNKNOWN:  2%
false success: 3.5%

If you only optimize apparent completion rate, Agent B looks better.

It is not.

A mature SLO system gives UNKNOWN its own budget.

For example:

verified_success_rate >= 92%
false_success_rate    <= 0.5%
UNKNOWN_rate          <= 8%

The objective is not to eliminate UNKNOWN blindly.

The objective is to reduce avoidable uncertainty without converting uncertainty into false confidence.

4. Reliability Is a Vector, Not One Number

Agent reliability usually needs multiple simultaneous objectives.

A useful vector is:

R = {
    verified_success,
    false_success,
    unknown_rate,
    latency,
    cost,
    side_effect_integrity,
    verifier_coverage
}

You should resist collapsing these immediately into one weighted score.

Why?

Because a weighted score can hide catastrophic trade-offs.

Example:

+3% verified success
-80% cost
+2.5% false success

A single utility function could still rank that as positive depending on the weights.

But the false-success ceiling may be a hard constraint.

A safer formulation is:

maximize verified_success

subject to:
    false_success <= 0.5%
    p95_latency   <= 180s
    p95_cost      <= $1.20
    verifier_coverage >= 99%

This mirrors the control-policy formulation from earlier steps in the series.

Reliability constraints come first.

Optimization happens inside them.

5. Define SLIs Before SLOs

An SLO is only useful if its underlying Service Level Indicator is measurable.

For agents, that means defining exactly how each metric is computed.

Example:

from dataclasses import dataclass
from enum import Enum


class Outcome(str, Enum):
    PASS = "PASS"
    FAIL = "FAIL"
    UNKNOWN = "UNKNOWN"


@dataclass(frozen=True)
class RunOutcome:
    run_id: str
    task_class: str
    outcome: Outcome
    externally_verified: bool
    false_success: bool
    latency_ms: int
    cost_usd: float

Then:

def verified_success_rate(rows: list[RunOutcome]) -> float:
    if not rows:
        return 0.0

    passed = sum(
        1
        for r in rows
        if r.outcome == Outcome.PASS and r.externally_verified
    )

    return passed / len(rows)

The exact denominator matters.

Are cancelled user requests included?

Are unsupported task classes included?

Are dependency-unavailable runs included?

Are retries counted as separate tasks or attempts?

Those choices must be explicit.

Otherwise teams can accidentally improve the metric by changing the population being measured.

6. SLOs Need Cohorts

A global SLO can hide local collapse.

Suppose the platform reports:

verified success = 96%

Looks excellent.

But the cohort breakdown is:

simple coding fixes      99%
research synthesis       98%
browser purchases        93%
production DevOps        78%

The aggregate is misleading.

The DevOps cohort may be unacceptable even if it is a small share of total traffic.

Useful cohort dimensions include:

• task class,
• risk class,
• tenant,
• repository size,
• tool family,
• model tier,
• workflow type,
• side-effect severity,
• verifier type,
• geographical/provider region,
• workload source.

A reliability contract should therefore support cohort-specific SLOs.

Example:

slos:
  coding.simple_fix:
    verified_success_min: 0.97
    false_success_max: 0.005

  coding.large_refactor:
    verified_success_min: 0.90
    false_success_max: 0.01

  devops.production_change:
    verified_success_min: 0.99
    false_success_max: 0.001
    duplicate_side_effect_max: 0.0001

The numbers are illustrative.

The architecture is the important part.

7. Separate Task Reliability From Component Reliability

Component metrics matter.

They are just not substitutes for task SLOs.

You may track:

router accuracy
retrieval recall@k
critic net correction
search pruning regret
model escalation rescue rate
verifier false-positive rate
browser tool error rate
lease expiration rate
retry amplification
queue deadline misses

These metrics help explain why the user-facing SLO moved.

The hierarchy should look like:

user-facing SLO
      ↓
workflow SLIs
      ↓
component metrics
      ↓
implementation counters

Not the other way around.

If a router metric improves but verified user outcomes get worse, the router metric is not the goal.

8. Error Budgets Turn Reliability Into a Decision Tool

Suppose the SLO is:

verified_success >= 99%

Then the allowed failure budget is:

1%

Over 100,000 eligible tasks, that means:

allowed failed outcomes = 1,000

If the platform consumes 700 of those failures halfway through the measurement window, the team has burned 70% of the error budget in 50% of the time.

That is actionable.

An error budget transforms reliability from:

“We had some incidents.”

into:

“We are consuming reliability allowance faster than the policy permits.”

That can change release behavior.

9. Burn Rate Matters More Than Remaining Budget Alone

A system can have plenty of budget remaining and still be in trouble.

Imagine a 30-day SLO window.

At day 2:

error budget consumed = 25%

That is alarming.

The raw remaining budget is 75%.

But the burn rate is far too high.

A simple burn-rate concept is:

actual_bad_event_rate
---------------------
allowed_bad_event_rate

If the SLO permits 1% bad outcomes and the recent window is running at 4%, then:

burn_rate = 4x

At 4x sustained burn, the budget will be exhausted much earlier than planned.

That should influence release policy.

10. Use Multiple Burn Windows

Short windows detect sharp regressions.

Long windows detect sustained degradation.

For example:

5-minute burn
1-hour burn
6-hour burn
24-hour burn
7-day burn

A sharp spike may indicate:

• a broken model deployment,
• a verifier regression,
• a tool outage,
• a bad prompt release,
• a routing policy bug.

A slow sustained burn may indicate:

• task-distribution shift,
• retrieval drift,
• model quality drift,
• gradual policy degradation,
• rising external API unreliability.

This connects directly to Step 23’s behavioral drift detection.

11. Different Failures Should Consume Different Budgets

Not all failures have equal severity.

A coding agent producing a patch that fails tests is different from a coding agent silently modifying the wrong repository.

A research agent returning UNKNOWN is different from fabricating a citation.

A browser agent timing out before purchase is different from purchasing twice.

You can model separate error budgets:

ordinary failure budget
false-success budget
side-effect-integrity budget
UNKNOWN budget
latency budget
cost budget

The critical ones may be zero-tolerance or near-zero-tolerance.

Example:

error_budgets:
  false_success:
    max_rate: 0.002

  duplicate_side_effect:
    max_rate: 0.0001

  unknown:
    max_rate: 0.08

  p95_latency_violation:
    max_rate: 0.05

This is more informative than one generic “error budget.”

12. Reliability Policy Should Change Release Policy

An error budget is useless if nothing happens when it burns.

A practical release policy could look like:

budget healthy
    ↓
normal feature velocity

budget elevated
    ↓
smaller canaries
more verification
reduced speculative complexity

budget critical
    ↓
freeze risky behavioral releases
prioritize reliability work
rollback recent regressions

budget exhausted
    ↓
no new capability rollout
except fixes that restore SLO compliance

The key idea is not bureaucracy.

It is making reliability trade-offs explicit.

13. Error Budgets Should Not Reward Under-Attempting

There is a subtle failure mode.

Suppose the system avoids difficult tasks by returning UNKNOWN immediately.

Its false-success rate may improve.

Its hard failure rate may improve.

But user value may collapse.

That is why you need multiple SLOs simultaneously.

Example:

verified_success >= 92%
false_success    <= 0.5%
UNKNOWN          <= 8%

The system cannot game reliability simply by refusing everything difficult.

14. Error Budgets Should Not Reward Easy-Traffic Routing

Another subtle failure mode is denominator manipulation.

Imagine the agent begins routing difficult tasks away from the measured workflow.

The measured success rate improves.

The actual platform did not.

Therefore the SLI definition should bind:

• eligibility rules,
• task-class definitions,
• cohort assignment,
• unsupported-task semantics,
• excluded events,
• attempt deduplication.

These should be versioned.

A metric definition is part of the behavioral contract.

15. Cost and Latency Need Budgets Too

An agent can hit reliability targets and still become operationally unusable.

Suppose a new search policy improves verified success from:

94% → 95%

but changes cost from:

$0.20 → $2.40

and p95 latency from:

40s → 240s

That may be unacceptable.

So define objectives such as:

p95_latency <= 120s
cost_per_verified_success <= $0.80

The important metric is often not average cost per run.

It is:

cost_per_verified_success

because cheap wrong answers are not useful.

16. Verification Coverage Needs Its Own Reliability Target

Suppose the platform reports:

verified success = 97%

But only 60% of tasks actually had strong external verification.

That is suspicious.

You need to know:

verification_coverage

For example:

strong_verifier_coverage >= 98%

for the task classes where strong verification is expected.

This protects against silently weakening the evidence standard.

It also connects directly to Step 22’s verifier degradation handling.

If verifier capacity drops, the system should not compensate by pretending verification is optional.

17. Verifier Reliability Is Part of the SLO System

The verifier itself can fail.

A permissive verifier creates false success.

An overly strict verifier inflates FAIL or UNKNOWN.

So monitor:

verifier_false_positive_rate
verifier_false_negative_rate
verifier_coverage
verifier_disagreement_rate

where ground truth is available through:

• deterministic checks,
• gold cases,
• human review,
• independent recomputation,
• downstream observed outcomes.

The verifier cannot be treated as infallible simply because it is called “the verifier.”

18. Use Risk-Tiered SLOs

Not every task deserves the same reliability requirement.

A low-risk summarization task may tolerate:

95% verified usefulness

A production infrastructure mutation may require:

99.9% verified correctness

with near-zero false success.

A useful risk model is:

LOW
MEDIUM
HIGH
CRITICAL

Then define increasingly strict requirements.

Example:

LOW
  cheaper verification allowed
  moderate UNKNOWN tolerated

MEDIUM
  stronger verifier required
  lower false-success ceiling

HIGH
  mandatory independent verification
  reduced speculative authority

CRITICAL
  deterministic preconditions
  explicit approval / fenced commit
  near-zero false-success budget

Risk classification should happen before execution, not after something goes wrong.

19. Reliability Should Influence the Scheduler

The Step 16 run scheduler and Step 21 platform scheduler should consume reliability state.

Example:

false-success budget burning too fast
        ↓
reserve more verifier capacity
reduce speculative fan-out
increase escalation threshold quality
route high-risk tasks to stronger verification

Or:

latency budget burning too fast
        ↓
reduce search width
prefer cached observations
use cheaper/faster route where reliability remains inside SLO

The important part is constraint ordering.

Do not fix latency by violating correctness SLOs.

20. Reliability Should Influence the Router

A router should not only optimize expected capability and cost.

It should also know whether a route is consuming error budget disproportionately.

Suppose:

local model route
verified success = 88%

frontier model route
verified success = 97%

If the workload SLO is 95%, the local route may be unsuitable for that cohort even if it is cheaper.

The router policy can therefore include:

route allowed only if predicted reliability >= cohort SLO

This connects the control plane directly to reliability policy.

21. Reliability Should Influence Search Depth

Search is not free.

But under high error-budget burn, additional search may be justified if it measurably rescues failures.

Conversely, if search depth increases cost without improving verified outcomes, the SLO system should expose that.

Useful metrics include:

verified_success_by_search_depth
false_success_by_search_depth
cost_per_verified_success_by_depth
latency_by_depth

This is more useful than saying:

deeper search is smarter.

22. Reliability Should Influence Critic Invocation

A critic may help some cohorts and hurt others.

Track:

wrong → correct
correct → correct
wrong → wrong
correct → wrong

Then connect critic policy to SLO impact.

If a critic causes too many correct → wrong transitions in a cohort, its use should be reduced there.

Again:

mechanisms must earn their cost through verified outcomes.

23. Error Budgets Connect Incidents to Roadmap Priority

Step 26 gave us incident forensics.

Now imagine the incident taxonomy shows:

routing failure          18 incidents
verifier false positive  4 incidents
stale state              9 incidents
selection failure       23 incidents

Raw incident count is not enough.

You also need error-budget impact.

Perhaps the four verifier incidents consumed more reliability budget than the 23 selection incidents because they caused harmful false success.

A useful prioritization table might be:

That is much better than fixing whatever incident happened most recently.

24. Define a Reliability Contract

A useful reliability contract can be versioned like any other policy artifact.

Example:

reliability_contract:
  version: reliability-v4

  scope:
    task_class: coding.patch
    risk: medium

  slos:
    verified_success_min: 0.95
    false_success_max: 0.005
    unknown_max: 0.06
    verifier_coverage_min: 0.98
    p95_latency_ms_max: 120000
    cost_per_verified_success_max: 0.80

  window:
    type: rolling
    duration_days: 30

  release_policy:
    warning_burn_rate: 2.0
    freeze_burn_rate: 5.0
    rollback_burn_rate: 10.0

The exact thresholds are domain-specific.

The important point is that the policy is explicit, diffable, replayable, and auditable.

25. Implement a Small Reliability Evaluator

You do not need an LLM to implement this.

from dataclasses import dataclass


@dataclass(frozen=True)
class ReliabilitySnapshot:
    verified_success: float
    false_success: float
    unknown_rate: float
    verifier_coverage: float
    p95_latency_ms: int
    cost_per_verified_success: float


@dataclass(frozen=True)
class ReliabilityContract:
    verified_success_min: float
    false_success_max: float
    unknown_max: float
    verifier_coverage_min: float
    p95_latency_ms_max: int
    cost_per_verified_success_max: float


def violations(
    snapshot: ReliabilitySnapshot,
    contract: ReliabilityContract,
) -> list[str]:
    out: list[str] = []

    if snapshot.verified_success < contract.verified_success_min:
        out.append("verified_success")

    if snapshot.false_success > contract.false_success_max:
        out.append("false_success")

    if snapshot.unknown_rate > contract.unknown_max:
        out.append("unknown_rate")

    if snapshot.verifier_coverage < contract.verifier_coverage_min:
        out.append("verifier_coverage")

    if snapshot.p95_latency_ms > contract.p95_latency_ms_max:
        out.append("p95_latency")

    if (
        snapshot.cost_per_verified_success
        > contract.cost_per_verified_success_max
    ):
        out.append("cost_per_verified_success")

    return out

This is ordinary software.

That is a feature.

26. Add Burn-Rate Evaluation

A simple burn-rate function can also remain deterministic.

def burn_rate(
    observed_bad_rate: float,
    allowed_bad_rate: float,
) -> float:
    if allowed_bad_rate <= 0:
        return float("inf") if observed_bad_rate > 0 else 0.0

    return observed_bad_rate / allowed_bad_rate

For a 99% success SLO:

allowed bad rate = 1%

If recent bad rate is 3%:

burn = 3x

The release controller can use this signal without asking a model to interpret it.

27. Do Not Let the Agent Change Its Own SLO

This boundary should be explicit.

The agent may observe:

• current burn rate,
• remaining budget,
• reliability mode,
• route restrictions.

But it should not be able to lower its own target because the target is inconvenient.

Bad architecture:

agent failing SLO
      ↓
agent decides SLO too strict
      ↓
agent lowers threshold
      ↓
problem disappears from dashboard

Good architecture:

external reliability policy
      ↓
agent receives constraints
      ↓
agent operates within constraints

Reliability authority belongs outside the optimized system.

28. Do Not Let the Verifier Define the SLO Alone

Similarly, the same verifier that judges production runs should not be the sole authority for whether the reliability target is met.

Why?

Because verifier drift can make the metric itself move.

Use independent checks where possible:

• deterministic acceptance tests,
• gold cases,
• human audits,
• secondary verifiers,
• downstream ground truth.

This is measurement-system integrity.

29. Reliability Windows Need Enough Data

Small samples are noisy.

Suppose a cohort has 10 tasks.

One failure changes the observed rate by 10 percentage points.

Do not build aggressive automatic rollback around tiny samples without confidence controls.

Useful safeguards include:

• minimum sample size,
• confidence intervals,
• paired comparisons,
• cohort pooling only when justified,
• long-window confirmation,
• severity-aware overrides for catastrophic failures.

A single critical false-success incident may justify immediate rollback even with small sample size.

So the policy needs both statistical and severity logic.

30. Availability SLOs Still Matter

Behavioral reliability does not replace infrastructure reliability.

You still need:

API availability
queue availability
worker availability
database availability
browser pool availability
model endpoint availability

The distinction is:

infrastructure SLOs
    measure ability to execute

behavioral SLOs
    measure quality of decisions and outcomes

Both matter.

Neither substitutes for the other.

31. Example: Coding Agent

Suppose a coding agent repairs failing tests.

Possible SLOs:

verified test-fix rate        >= 95%
false-success rate            <= 0.5%
wrong-repository mutation     = 0
p95 latency                   <= 180s
cost per verified fix         <= $1.00

If false success burns too quickly:

increase mandatory test coverage
require diff-level verification
reduce critic-only acceptance
route risky refactors to stronger verification

If latency burns but correctness is healthy:

reduce search width
reuse cached repository evidence
parallelize independent diagnostics

The corrective action depends on which SLO is burning.

32. Example: Research Agent

Possible SLOs:

citation-supported claims >= 98%
unsupported claims        <= 0.2%
UNKNOWN                   <= 10%
p95 latency               <= 90s

If unsupported-claim budget burns:

increase primary-source requirement
strengthen claim-source binding
reduce answer synthesis when evidence is weak

If UNKNOWN burns:

improve retrieval coverage
add targeted source discovery
fix stale indexes

Do not solve both by simply using a bigger model.

33. Example: Browser Agent

Possible SLOs:

verified transaction success >= 99%
duplicate transaction rate   <= 0.01%
ambiguous completion          <= 0.2%

If ambiguous completion burns:

strengthen postcondition checks
record transaction identifiers
query authoritative account state after timeout

If duplicate side effects burn:

fix idempotency
strengthen operation IDs
improve fencing / commit gateway

That is a distributed-systems problem, not a prompt problem.

34. Example: DevOps Agent

Possible SLOs:

verified safe change >= 99.5%
unauthorized mutation = 0
rollback failure      <= 0.1%
false-success         <= 0.05%

When the error budget degrades:

reduce autonomous authority
increase approval requirements
reserve more verification capacity
restrict high-risk tool routes

Reliability state can therefore directly control authority.

35. Reliability Mode as Runtime State

You can expose a coarse reliability mode:

HEALTHY
WATCH
CONSTRAINED
FREEZE

For example:

from enum import Enum


class ReliabilityMode(str, Enum):
    HEALTHY = "HEALTHY"
    WATCH = "WATCH"
    CONSTRAINED = "CONSTRAINED"
    FREEZE = "FREEZE"

The platform scheduler can then consume it.

Example:

HEALTHY
  normal rollout

WATCH
  smaller canaries
  more shadow traffic

CONSTRAINED
  restrict risky routes
  reserve verifier capacity
  reduce speculative work

FREEZE
  no new behavioral capability releases
  reliability fixes only

This connects SLOs to real operational behavior.

36. Error Budgets Prevent Endless Reliability Work Too

Error budgets are not only a mechanism for stopping feature releases.

They also protect teams from trying to eliminate every possible failure.

If reliability is comfortably inside the agreed SLO, the platform may not need another layer of complexity.

That is consistent with the entire series.

Advanced mechanisms are justified by measured failure, not architectural ambition.

If the reliability contract is healthy, adding another critic, search layer, verifier, or coordinator may make the system worse.

The error budget gives permission to stop hardening when the evidence says reliability is good enough.

37. Reliability Targets Must Be Economically Realistic

A target like:

99.9999% verified success

may be meaningless if achieving it requires:

• ten frontier-model calls,
• three independent verifiers,
• human review,
• five-minute latency,
• $50 per task.

Reliability exists inside an economic envelope.

The real question is:

What level of verified reliability is required for the task’s risk and value, at an acceptable cost and latency?

That is a product and engineering decision.

Not a model benchmark decision.

38. Reliability Targets Should Be Derived From Consequence

A useful heuristic is:

higher consequence
      ↓
higher verification requirement
      ↓
lower false-success tolerance
      ↓
more conservative authority

This is better than assigning one global target to every workflow.

For low-consequence tasks, the cheapest reliable architecture may be enough.

For consequential tasks, the system may need:

• independent verification,
• approval gates,
• stronger provenance,
• stricter replay guarantees,
• fenced side effects,
• lower error budgets.

39. Make SLO Breaches Explainable

A dashboard should not merely say:

SLO FAILED

It should explain:

SLO: verified_success >= 95%
observed: 91.8%
window: 6h
burn rate: 3.2x

largest contributing cohorts:
  large_repo_refactor: -6.4 pp
  browser_tool_v7:     -4.1 pp

largest failure classes:
  selection failure:   31%
  stale state:         27%
  verifier unknown:    18%

recent changes:
  router v12 → v13
  browser tool v6 → v7

Now the error budget becomes an entry point into Step 26 incident forensics.

40. SLOs Need Versioned Definitions

If the metric definition changes, historical comparison can become invalid.

Version:

• eligibility rules,
• outcome semantics,
• verifier requirements,
• false-success definition,
• UNKNOWN semantics,
• cohort rules,
• measurement windows.

Example:

reliability-contract-v4

should be stored alongside each evaluation snapshot.

Otherwise a chart can appear to improve simply because the definition changed.

41. Store Reliability Snapshots as Evidence

A reliability snapshot should be reproducible.

Example:

@dataclass(frozen=True)
class SLOEvaluation:
    evaluation_id: str
    contract_version: str
    cohort: str
    window_start: str
    window_end: str
    eligible_runs: int
    verified_success: float
    false_success: float
    unknown_rate: float
    p95_latency_ms: int
    cost_per_verified_success: float
    burn_rates: dict[str, float]
    release_versions: tuple[str, ...]

This makes later incident analysis much stronger.

You can ask:

Which release versions consumed this error budget?
Which cohorts drove it?
Which failure classes dominated?

42. Test the SLO System Itself

The measurement system can fail.

Test cases should include:

Denominator changes

Ensure unsupported tasks are not silently excluded differently between versions.

Duplicate attempts

One user task retried five times should not necessarily become five independent failures.

Verifier outage

Ensure missing verification does not become PASS.

Late evidence

Ensure a delayed postcondition can reconcile an earlier UNKNOWN correctly.

Cohort reassignment

Ensure tasks do not migrate between cohorts silently after execution.

Clock/window boundaries

Ensure rolling windows do not double-count or lose events.

Release attribution

Ensure mixed-version distributed runs are attributed correctly.

Reliability metrics are software.

They need tests too.

43. Failure Injection Should Consume the Expected Budget

If you deliberately inject a known failure mode, the SLO machinery should react predictably.

Examples:

inject stale repository snapshot
      ↓
state-uncertainty / stale-state failures rise
      ↓
verified success falls
      ↓
relevant burn rate rises

Or:

make verifier permissive
      ↓
false-success incidents rise
      ↓
false-success budget burns rapidly

If the dashboard does not reflect the injected failure, the measurement system is incomplete.

44. Error Budgets Create a Reliability Feedback Loop

We now have a full loop:

production runs
      ↓
verified outcomes
      ↓
SLIs
      ↓
SLO evaluation
      ↓
error-budget burn
      ↓
release / scheduler / authority policy
      ↓
incidents and drift investigation
      ↓
remediation
      ↓
shadow / canary
      ↓
production

This is where the earlier advanced-agent mechanisms start behaving like a real engineering platform rather than a pile of orchestration patterns.

45. The Most Important Metric Is Still the External Outcome

Throughout this series we have introduced:

• routers,
• search,
• critics,
• memory,
• multi-agent orchestration,
• dynamic budgets,
• uncertainty decomposition,
• Expected Value of Information,
• speculative concurrency,
• distributed coordination,
• global scheduling,
• circuit breakers,
• drift detection,
• release engineering,
• replay,
• incident forensics.

Each mechanism has internal metrics.

But those metrics exist to explain and improve externally verified outcomes.

The hierarchy remains:

external verified outcome
        ↓
reliability SLO
        ↓
workflow metrics
        ↓
component metrics

Never reverse that hierarchy.

46. A Minimal Production Reliability Stack

A practical implementation does not need to begin with a giant SRE platform.

Start with:

1. explicit PASS / FAIL / UNKNOWN
2. false-success measurement
3. one verified-success SLO
4. one false-success ceiling
5. one latency target
6. one cost target
7. cohort breakdown
8. rolling error-budget burn
9. release freeze / rollback rules
10. incident attribution

Then add complexity only when the failure evidence demands it.

That is enough to make reliability operational rather than rhetorical.

47. What Not to Do

Do not define reliability as:

model confidence > 0.8

Do not define reliability as:

agent completed without exception

Do not define reliability as:

critic approved result

Do not define reliability as:

majority of agents agreed

Do not define reliability as:

HTTP 200

Do not let the candidate agent lower its own target.

Do not let UNKNOWN become fake PASS.

Do not hide false success inside one aggregate metric.

Do not compare cohorts with different risk semantics as if they were identical.

Do not keep shipping risky behavioral releases while the error budget is burning uncontrollably.

And do not keep adding reliability machinery forever when the measured reliability is already sufficient for the task.

48. The Bigger Architectural Shift

The deeper idea is not really about SLO dashboards.

It is about turning reliability into a control input.

Once reliability is explicit, the platform can make better decisions about:

• which models may serve which task classes,
• how much search is justified,
• when escalation is required,
• how much verifier capacity to reserve,
• whether speculative work should be reduced,
• whether a behavioral release may proceed,
• whether authority should be constrained,
• which reliability fix deserves engineering time.

The platform stops asking:

Is the agent good?

and starts asking:

Is this cohort meeting its verified reliability contract inside the agreed cost, latency, and risk envelope?

That is a much more useful engineering question.

49. Final Architecture

The reliability layer now sits above the mechanisms developed throughout the series:

user task
   ↓
platform admission
   ↓
run control policy
   ↓
routing / search / tools / memory / critics
   ↓
execution
   ↓
external verification
   ↓
PASS / FAIL / UNKNOWN
   ↓
SLIs
   ↓
SLO evaluation
   ↓
error-budget burn
   ↓
release / scheduling / authority policy
   ↓
incident forensics / remediation

The important property is feedback.

Reliability evidence changes architecture and release decisions.

The architecture is no longer allowed to grow independently of measured outcomes.

50. Final Rule

The rule for agent reliability is simple:

Define what good enough means before production failure forces you to invent the definition after the fact.

Then measure it with external evidence.

Give false success its own ceiling.

Treat UNKNOWN honestly.

Track cost and latency alongside correctness.

Slice by risk and task class.

Turn failures into error-budget burn.

And let the burn rate decide when the platform should keep expanding capability and when it should stop and become more reliable.

That is how agent reliability becomes an engineering discipline rather than a collection of post-incident opinions.

What Comes Next?

SLOs tell us whether reliability is acceptable.

Error budgets tell us whether the team is consuming that reliability too quickly.

The next problem is deciding which reliability work should happen first when several components are contributing to the burn.

That leads naturally to the next stage:

Reliability economics and risk-based prioritization: how do you decide whether to spend engineering effort on model quality, verification, routing, tooling, infrastructure, or process when every improvement has a different cost and expected reduction in failure risk?