Your Agent Changed the World. What Happens When Step Two Fails? Build Transactions, Compensation and Reconciliation
An agent can reason correctly, choose the right tool, pass authorization, and still leave the world in a broken intermediate state.
The failure does not require a hallucination.
It can happen because step one succeeds and step two does not.
create deployment ✓
update DNS ✓
run migration ✓
start application ✗
notify customer not attempted
Now what?
A naive agent architecture often has only two ideas:
success
failure
But the external world may already have changed.
There may be no global rollback button.
There may not even be one authoritative system that knows whether the last request succeeded.
This is where agent engineering stops looking like prompt engineering and starts looking like distributed transaction engineering.
The core principle for this chapter is:
Rollback is not the inverse of every action.
The correct runtime needs explicit semantics for preparation, commitment, verification, compensation and reconciliation.
The Search Problem: “How Do I Roll Back an AI Agent After a Partial Failure?”
Consider an agent that has to onboard a customer.
It might need to:
- create a tenant in the application database,
- create a billing customer,
- provision cloud resources,
- create DNS records,
- configure access control,
- send a welcome email.
Those actions may span six independent systems.
There is no database transaction wrapping all of them.
If step four fails, the system must answer several different questions:
What definitely happened?
What may have happened?
What can be undone?
What can only be compensated?
What is irreversible?
What commitments are still active?
What state must be reconciled before retrying?
That is not one error-handling branch.
It is a transaction model.
Why ACID Does Not Magically Extend Across Tools
Inside one relational database we may have:
BEGIN
UPDATE ...
INSERT ...
COMMIT
The database coordinates atomicity.
An agent workflow commonly crosses boundaries like:
GitHub
Stripe
AWS
Kubernetes
email provider
browser session
internal database
human approval queue
Those systems do not share one transaction coordinator.
Even if two systems each offer transactions internally, their transactions are not automatically atomic together.
The agent therefore needs to reason about a sequence of durable external effects.
But the model should not invent the transaction semantics.
Those semantics belong in the execution platform.
First Principle: Classify Side Effects Before You Execute Them
Not every action has the same recovery properties.
A useful minimum classification is:
REVERSIBLE
COMPENSATABLE
IRREVERSIBLE
UNKNOWN
Reversible
A reversible operation has a trusted inverse that restores the relevant prior state closely enough for the system’s correctness requirements.
Example:
set feature flag ON
may be reversible by:
set feature flag OFF
provided no irreversible downstream effects occurred while the flag was enabled.
That last condition matters.
The inverse command existing does not prove the whole effect is reversible.
Compensatable
A compensatable action cannot literally be erased, but another action can reduce or neutralize its business effect.
Example:
charge card
may be compensated by:
issue refund
The charge still occurred.
The refund is a second real-world event.
Accounting records, notifications and timing all remain part of history.
So:
refund != undo(charge)
Irreversible
Some effects cannot reliably be undone.
Examples may include:
send email
publish public message
delete unrecoverable external data
trigger irreversible physical action
reveal secret to recipient
Once executed, the correct recovery action may be explanation, escalation or mitigation rather than rollback.
Unknown
This is the most dangerous category.
Suppose the agent calls an external API and receives a timeout.
POST /create-order
↓
network timeout
The timeout does not prove failure.
The remote system may have committed the order and lost the response.
The operation is now:
OUTCOME_UNKNOWN
Retrying blindly can duplicate the side effect.
A Production Transaction Lifecycle
For consequential external effects, use an explicit lifecycle:
prepare
↓
validate
↓
commit
↓
observe authoritative result
↓
verify postcondition
↓
complete
And on failure:
failure / ambiguity
↓
classify current reality
↓
compensate if appropriate
↓
verify compensation
↓
reconcile remaining state
↓
complete / escalate / UNKNOWN
The important detail is that recovery has verification too.
A compensation request is not proof that compensation succeeded.
Prepare Before Commit
Many failures are cheaper to prevent than repair.
The prepare phase should gather everything that can be checked before mutation.
For example:
intent current?
authority valid?
state fresh?
competence sufficient?
verifier available?
dependent resources available?
idempotency key assigned?
operation identity assigned?
compensation plan known?
commitments understood?
A simplified DTO might look like:
from dataclasses import dataclass
from enum import Enum
class RecoveryClass(str, Enum):
REVERSIBLE = "reversible"
COMPENSATABLE = "compensatable"
IRREVERSIBLE = "irreversible"
UNKNOWN = "unknown"
@dataclass(frozen=True)
class PreparedOperation:
operation_id: str
intent_id: str
intent_version: int
action_type: str
target_ref: str
state_version: str
authority_ref: str
idempotency_key: str
recovery_class: RecoveryClass
compensation_operation_type: str | None
expected_postcondition: str
The important property is not this exact Python shape.
The important property is that recovery semantics are known before a consequential action is allowed to commit.
Commit With Stable Operation Identity
Every consequential logical operation should have a stable identity.
logical operation
↓
operation_id = OP-123
↓
physical attempt 1
physical attempt 2
physical attempt 3
Do not assign a new business operation identity just because a network retry occurred.
That enables idempotency.
For example:
operation_id: OP-123
idempotency_key: customer-827:create-billing-account:v1
If the provider supports native idempotency keys, use them.
If it supports conditional writes, use them.
If it supports compare-and-set or version preconditions, use them.
The model should not simulate guarantees the external system can enforce exactly.
Exactly Once Is Usually the Wrong Promise
Distributed systems make exactly-once side effects difficult.
A more useful design is:
at-least-once delivery
+
idempotent logical operation
+
authoritative reconciliation
Suppose the worker sends:
create invoice INV-927
and crashes after the remote service commits it but before recording success.
On restart, the platform should not ask the model:
“Do you think the invoice probably exists?”
It should query authoritative state using the stable operation identity.
lookup INV-927
Then transition based on reality.
Postcondition Verification Is Part of the Transaction
An API returning 200 OK is not always the business outcome you intended.
The agent may request:
deploy version 42
and receive success from the deployment API.
The actual postcondition may be:
version 42 serving traffic
health checks passing
error rate below threshold
schema compatible
So the mutation pipeline should be:
request accepted
↓
external mutation
↓
read authoritative state
↓
verify desired postcondition
The verifier is external to the action generator.
A tool response saying success=true is evidence about the request, not necessarily proof of the intended outcome.
Compensation Is a New Transaction
This is one of the most important rules in the chapter.
Compensation is not metadata attached to the failed action. It is another consequential workflow with its own authority, identity, retries and verification.
If the original operation was:
charge €100
and the compensation is:
refund €100
then the refund should have its own:
operation_id
idempotency_key
authority decision
state preconditions
external result
postcondition verification
provenance
A minimal record might be:
@dataclass(frozen=True)
class CompensationRequest:
compensation_id: str
original_operation_id: str
compensation_type: str
reason: str
expected_postcondition: str
The connection to the original operation is explicit.
But the compensation is not treated as if the original operation never happened.
Compensation Can Fail Too
Imagine:
reserve inventory ✓
charge payment ✓
create shipment ✗
The workflow decides to compensate:
release inventory ✓
refund payment ✗
Now the system is still inconsistent.
The correct state is not:
FAILED
It is more specific:
COMPENSATION_FAILED
RECONCILIATION_REQUIRED
This matters operationally because cleanup work must remain visible and durable.
It must not disappear because the original customer-facing workflow has already failed.
Reconciliation Is Different From Compensation
Compensation asks:
What corrective action should we take?
Reconciliation asks:
What is actually true now?
That distinction is crucial after ambiguous outcomes.
Suppose:
create DNS record
↓
timeout
Before deciding whether to retry or compensate, reconcile:
read authoritative DNS state
↓
record exists? yes / no / unknown
record has expected value? yes / no / unknown
Then choose the next transition.
Reconciliation is therefore often:
observe first
act second
rather than:
failed request
→ immediately issue opposite request
Use a Transaction Ledger
For multi-system workflows, persist a transaction ledger.
For example:
@dataclass
class OperationRecord:
operation_id: str
workflow_id: str
intent_id: str
intent_version: int
action_type: str
target_ref: str
recovery_class: RecoveryClass
state: str
idempotency_key: str
attempt_count: int
external_reference: str | None
compensation_operation_id: str | None
verification_ref: str | None
Possible states:
PREPARED
COMMITTING
COMMITTED
VERIFYING
VERIFIED
OUTCOME_UNKNOWN
COMPENSATION_REQUIRED
COMPENSATING
COMPENSATED
COMPENSATION_FAILED
RECONCILIATION_REQUIRED
RECONCILED
IRREVERSIBLE_EFFECT
Do not compress these into one boolean called success.
Connect the Ledger to the Durable Workflow
Step 39 gave us a durable workflow runtime.
The operation ledger fits inside it naturally:
workflow
│
├── operation A ── VERIFIED
│
├── operation B ── VERIFIED
│
├── operation C ── OUTCOME_UNKNOWN
│ │
│ └── reconciliation activity
│
└── protected cleanup path
The workflow can sleep, restart, migrate workers or wait for a provider without losing the transaction state.
The model does not need to remember which effects occurred.
That is durable control-plane data.
Side-Effect Graphs Are Better Than Flat Lists
Real workflows are often not simple sequences.
Imagine:
create tenant
├── create billing account
├── create storage
└── create DNS
↓
deploy service
The correct compensation order may depend on this graph.
For example, you may need to stop traffic before deleting storage.
So model side-effect dependencies explicitly:
operation A ENABLES B
operation B DEPENDS_ON A
operation C MUST_COMPENSATE_BEFORE A
This lets the workflow calculate a safe recovery order.
Do not assume reverse chronological order is always correct.
Saga-Like Semantics Without Pretending Everything Is Reversible
A useful conceptual model is a saga:
T1 → T2 → T3 → T4
with compensations:
C1 ← C2 ← C3
But there are two important corrections for agent systems.
First, not every transaction step has a valid compensation.
Second, the compensation order may depend on current external state rather than merely being a predefined reverse list.
So the agent platform should represent recovery explicitly rather than assuming:
compensate(step_n ... step_1)
will always restore correctness.
Irreversible Effects Change Planning
If an action is irreversible, that information should affect the plan before execution.
For example:
prepare draft email
verify recipient
verify content
human approval if required
send email
The irreversible step should move toward the end of the workflow whenever practical.
This is a general design principle:
Delay irreversible side effects until uncertainty has been reduced as far as economically reasonable.
That connects directly to earlier chapters on verification, uncertainty and Expected Value of Information.
Before an irreversible action, buying one more authoritative observation may have unusually high value.
Order Work by Recoverability
Suppose two plans produce the same successful outcome.
Plan A:
irreversible action
reversible action
verification
Plan B:
verification
reversible action
irreversible action
postcondition verification
All else equal, Plan B has a better recovery profile.
A planner can therefore consider recoverability as a constraint or decision feature.
Do not let the model invent numeric precision here.
A deterministic risk class is often enough:
LOW_RECOVERY_RISK
MODERATE_RECOVERY_RISK
HIGH_RECOVERY_RISK
IRREVERSIBLE
Commitments Survive Partial Failure
Step 38 introduced commitments.
Transactions interact with them directly.
Suppose a workflow reserves a deployment window and creates a maintenance notification.
Then deployment fails.
The failed workflow cannot simply terminate.
It must ask:
Is the maintenance window still active?
Can it be released?
Was the customer notified?
Does a follow-up commitment now exist?
The transaction recovery path therefore includes commitment resolution:
satisfy
release
transfer
compensate
reconcile
Cancellation Does Not Erase Transactions
Step 37 established that intent can be cancelled or superseded.
Suppose cancellation arrives after two of five side effects have committed.
intent v7 ACTIVE
operation A VERIFIED
operation B COMMITTED
operation C not started
cancel intent v7
Correct behavior:
fence future business mutations
stop C, D, E
inspect A and B
resolve commitments
compensate or reconcile if policy requires
verify final state
Incorrect behavior:
cancel workflow
forget everything
Cancellation may create more mandatory work, not less.
Cleanup and reconciliation need protected capacity.
Compensation Must Respect Current Intent Too
There is a subtle problem.
Suppose intent v7 is cancelled and a compensation is scheduled.
Then intent v8 arrives and explicitly wants to preserve one of the effects created by v7.
Blindly executing the old compensation could now be wrong.
So compensation should be bound to:
original operation
current recovery policy
current intent lineage
current external state
Before committing compensation, revalidate whether the compensation is still appropriate.
This is the same temporal-consistency principle from Step 36 applied to recovery work.
Human Approval Can Be Part of Recovery
Some recovery decisions should not be autonomous.
For example:
customer charged
inventory committed
shipment creation failed
The allowed recovery choices may be:
refund customer
hold order for manual fulfillment
substitute inventory source
escalate to support
Those are business decisions, not merely technical inverses.
The workflow can prepare evidence and options while authority remains with a human.
Remember the Step 29 distinction:
authorization
≠
verification
A human can authorize a refund.
The platform must still verify that the refund actually occurred.
Ambiguous Effects Should Block Unsafe Follow-Up
If the state of an operation is unknown, downstream work may need to stop.
For example:
payment status UNKNOWN
should usually prevent:
ship order
until reconciliation establishes whether payment succeeded.
That means operation dependencies can carry predicates such as:
requires VERIFIED
requires NOT_COMMITTED
requires RECONCILED
This prevents the workflow from treating uncertainty as failure or success by convenience.
UNKNOWN Is Not a Temporary Embarrassment
The platform should be comfortable ending a decision cycle with:
OUTCOME_UNKNOWN
RECONCILIATION_REQUIRED
HUMAN_REQUIRED
if authoritative reality cannot currently be established.
The dangerous alternative is fabricating certainty so the workflow can continue.
Example: GitHub Deployment Workflow
Consider an agent that prepares and deploys a code change.
1. create branch
2. commit code
3. open PR
4. merge PR
5. deploy release
6. verify production
Recovery classes may be:
create branch reversible
commit code reversible-ish / preserved history
open PR compensatable by close
merge PR not cleanly reversible
production deploy compensatable by new rollback release
external message potentially irreversible
If production verification fails after merge, the agent should not pretend the Git history can simply be erased.
The recovery may be:
create corrective commit
open rollback PR
verify rollback
redeploy
verify production
record incident
That is compensation and forward recovery, not literal rollback.
Example: Browser Purchase
A browser agent might:
select item
enter delivery address
submit payment
receive timeout
The timeout creates an ambiguous side effect.
Do not:
click Buy again
Instead:
query order history
query payment state
match stable transaction identifiers
reconcile
Only then determine whether a retry is valid.
This is why browser automation needs transaction semantics just as much as backend APIs.
Example: Infrastructure Automation
An infrastructure agent might:
create database
create secrets
create service
attach load balancer
update DNS
If DNS update fails, deleting everything may be worse than finishing the deployment later.
Recovery policy may choose:
KEEP_AND_RETRY
COMPENSATE_PARTIAL
ROLL_FORWARD
ESCALATE
The correct choice depends on commitments, risk, cost, freshness and operational policy.
Do not encode every failure as “delete what we created.”
Roll Forward Is Often Safer Than Roll Back
Many systems are easier to repair by completing a corrected forward transition.
Examples:
bad database migration
→ corrective migration
bad deployment
→ new known-good deployment
incorrect configuration
→ corrected configuration version
This preserves history and avoids pretending the previous world can be reconstructed exactly.
A useful recovery policy therefore includes:
ROLL_FORWARD
COMPENSATE
REVERSE
RECONCILE
HUMAN_ESCALATE
NO_AUTOMATED_RECOVERY
Make Recovery Policy Explicit
A platform can define recovery policy separately from model reasoning.
@dataclass(frozen=True)
class RecoveryPolicy:
action_type: str
recovery_class: RecoveryClass
automatic_compensation_allowed: bool
human_approval_required: bool
max_compensation_attempts: int
reconciliation_required_after_timeout: bool
The model may suggest a recovery strategy.
It should not be able to rewrite the safety constraints governing its own failed action.
Compensation Authority Should Be Narrow
A common mistake is to grant a cleanup worker broad privileges because it is “only undoing things.”
But compensation can be as consequential as the original action.
A refund changes money.
Deleting a resource destroys state.
Revoking access changes authorization.
Therefore:
original action authority
and:
compensation authority
should be explicit and independently scoped.
Recovery Has Its Own Budget
Earlier chapters protected verification capacity.
Durable systems should also protect reconciliation and cleanup capacity.
If the platform is overloaded and sheds all work indiscriminately, it can strand half-completed workflows.
A useful resource split is:
new work
speculative work
verification work
reconciliation work
critical compensation work
Under severe pressure:
shed speculation first
slow new work
preserve verification
preserve reconciliation
preserve required compensation
Otherwise overload can convert temporary failures into lasting inconsistency.
Transaction State Belongs in Replay and Provenance
Step 25 introduced deterministic replay and provenance.
Every consequential operation should record enough information to reconstruct:
why action was attempted
which intent authorized it
which state version was observed
which operation identity was used
which provider/system received it
what response occurred
what authoritative state was later observed
what verifier concluded
whether compensation occurred
what final state remained
That enables incident investigators to distinguish:
request failed
response was lost
remote commit succeeded
verification failed
compensation failed
reconciliation never ran
Those are radically different incidents.
Transaction Events for Trajectory Observability
Useful structured events include:
operation.prepared
operation.commit_started
operation.response_received
operation.outcome_unknown
operation.postcondition_verified
operation.compensation_requested
operation.compensation_started
operation.compensation_verified
operation.reconciliation_started
operation.reconciliation_completed
operation.human_escalated
operation.irreversible_effect_recorded
Each event should carry stable identifiers rather than relying on prose logs.
Metrics That Matter
Do not optimize for “few rollbacks” in isolation.
That can simply mean failures are being ignored.
Track metrics such as:
partial-commit rate
ambiguous-outcome rate
reconciliation latency
compensation success rate
compensation failure rate
irreversible-effect rate
duplicate-side-effect rate
idempotency conflict rate
postcondition failure rate
recovery human-escalation rate
orphaned-operation count
mean time to consistent state
One especially useful metric is:
time_to_consistent_state
because user-facing failure may happen before the system is operationally clean again.
Recovery Quality Matters More Than Workflow Completion
A workflow that reports FAILED in ten seconds but leaves three external systems inconsistent is not necessarily healthier than one that spends five minutes reconciling and reaches a known safe state.
So terminal states should distinguish:
FAILED_CLEAN
FAILED_COMPENSATED
FAILED_RECONCILED
FAILED_WITH_IRREVERSIBLE_EFFECT
FAILED_RECONCILIATION_REQUIRED
UNKNOWN
This makes operational reality visible.
Failure Injection Is Mandatory
Transaction recovery should be tested by deliberately failing at every meaningful boundary.
For example:
crash before request
crash after send before response
provider commits then times out
provider rejects duplicate idempotency key
verification service unavailable
compensation provider unavailable
compensation commits then response lost
intent cancelled during compensation
state changes before compensation
worker loses lease during reconciliation
human approval expires during recovery
The invariant is not:
workflow never fails
It is:
Every consequential side effect reaches a known verified state, a protected recovery path, or an explicit unresolved state that cannot silently progress.
Deterministic Transaction Invariants
Some checks should never depend on an LLM.
For example:
def may_execute_downstream(parent: OperationRecord) -> bool:
return parent.state in {"VERIFIED", "RECONCILED"}
def may_retry_unknown(operation: OperationRecord) -> bool:
return operation.state != "OUTCOME_UNKNOWN"
In reality, retryability can be more nuanced.
But the principle stands:
Known exact state-machine facts belong in deterministic control code.
Do You Actually Need a Transaction Layer?
Not every agent does.
If the system only:
reads documents
produces suggestions
writes drafts
and never performs consequential external mutations, a complex compensation engine may be unnecessary.
Start with the simplest boundary that matches the risk.
A useful progression is:
read-only agent
↓
typed side effects + postcondition verification
↓
idempotent operations
↓
durable operation ledger
↓
compensation / reconciliation
↓
multi-system recovery graph
Add each layer because measured failure requires it.
What This Adds to the Architecture
The series now has:
intent lifecycle
↓
goals and commitments
↓
durable workflow
↓
transaction semantics
↓
external side effects
The durable workflow answers:
Where are we in the process?
The transaction layer answers:
What has actually changed in the world, and what must happen if only part of the intended change succeeded?
That is a different question.
Both are necessary for consequential autonomy.
The Deeper Principle
Agent systems are often described as systems that can “take actions.”
That description is incomplete.
Production systems need to know:
which action was intended
which action was authorized
which action was attempted
which action definitely happened
which action may have happened
which outcome was verified
which obligation remains
which recovery is possible
which recovery was attempted
which final state is now true
That is the difference between action generation and operational control.
The final principle for this step is:
Do not design agents around successful action sequences. Design them around partial success, ambiguity and recovery.
The next stage is trust.
A system can now reason, coordinate, survive failure, reconcile external reality and recover from partial transactions.
But it still consumes instructions and evidence from users, tools, retrieved documents, web pages, other agents and generated code.
Those sources do not deserve equal authority.
The next question is therefore:
What should your agent trust?
