How Do You Move a Running Agent Between Workers Without Losing Meaning? Build Portable Execution State and Safe Handoff

A long-running agent starts on one worker.

Then reality intervenes.

The GPU is reclaimed.

A provider becomes unhealthy.

A region loses capacity.

The browser pool is drained.

A more suitable model becomes available.

The scheduler decides the run should move.

So you checkpoint the agent and resume it somewhere else.

Simple.

Except it is not.

An agent run is not merely a Python object containing a few messages.

By the time a serious run has been executing for several minutes, it may have accumulated:

• a task interpretation,
• retrieved evidence,
• tool observations,
• working memory,
• search branches,
• rejected candidates,
• budget consumption,
• verifier evidence,
• external side effects,
• pending operations,
• approval state,
• lease ownership,
• fencing epochs,
• placement assumptions,
• capability and authority decisions,
• environment identity,
• release identity,
• and provenance links connecting all of those things.

If we move only the conversation transcript, we have not moved the run.

We have created a new run that merely remembers some text from the old one.

That distinction matters.

Step 34 gave us capability-aware placement across heterogeneous execution targets.

Now we need to answer the next question:

How can a running agent move between workers, models, providers or regions without silently changing the meaning, evidence, ownership or authority of the run?

The core rule for this post is:

A task may move. Its meaning, evidence, and authority must not move implicitly.

The Search Problem: “How Do I Resume an AI Agent on Another Worker Safely?”

A naive checkpoint often looks like this:

checkpoint = {
    "messages": messages,
    "current_step": step,
}

That may be enough for a toy chatbot.

It is not enough for a production agent that can observe changing external state or perform side effects.

Suppose a coding agent has:

1. read repository commit A,
2. generated patch P,
3. run tests against workspace W,
4. received verifier evidence V,
5. requested approval for the exact hash of P,
6. then loses its worker before commit.

The replacement worker must not simply deserialize the patch and continue.

It needs to know:

Was P verified against the same repository state?
Is approval still valid for this exact artifact?
Has another worker already committed the operation?
Has the repository changed since verification?
Does the replacement worker have the same authority?
Is its tool contract compatible?
Is the verifier evidence still applicable?

Without those answers, “resume” can become duplicated mutation, stale execution, authority escalation or verifier misbinding.

Portable execution state is therefore not serialization.

It is a continuation contract.

1. Separate Logical State From Physical Worker State

The first architectural distinction is:

logical run state
    ≠
worker process state

A worker may contain:

• open sockets,
• Python objects,
• GPU tensors,
• browser handles,
• temporary directories,
• model KV caches,
• file descriptors,
• local locks,
• process IDs.

Most of those are not portable.

The logical run should instead be reconstructable from durable references.

For example:

from dataclasses import dataclass

@dataclass(frozen=True)
class ExecutionCheckpoint:
    run_id: str
    checkpoint_id: str
    release_id: str
    state_schema_version: str
    task_state_ref: str
    observation_manifest_ref: str
    workspace_snapshot_ref: str | None
    memory_snapshot_ref: str | None
    search_state_ref: str | None
    verifier_evidence_ref: str | None
    operation_ledger_ref: str
    authority_context_ref: str
    budget_ledger_ref: str
    placement_ref: str
    owner_epoch: int
    checkpoint_hash: str

The checkpoint should point to durable, versioned artifacts rather than assume the replacement worker can reconstruct hidden process memory.

2. Define Resumability Classes

Not every step is equally resumable.

A useful runtime can classify execution points explicitly.

R0 — not resumable
R1 — restart from previous durable boundary
R2 — resume from deterministic state
R3 — resume with recorded observations
R4 — resume after ownership transfer
R5 — resume after external side effects with reconciliation

This classification prevents the scheduler from assuming every interruption can be solved by loading a checkpoint.

Example

A pure planning step might be R2.

A research step using recorded web observations might be R3.

A coding task in an isolated workspace may be R4.

A task that sent an external API request and timed out before receiving confirmation may require R5 reconciliation.

Sometimes the correct state is simply:

MIGRATION_UNSAFE

That is better than pretending portability exists where it does not.

3. Checkpoint at Semantic Boundaries

Do not checkpoint at arbitrary process locations if the state cannot be interpreted safely.

Prefer semantic boundaries such as:

observation completed
candidate produced
candidate verified
approval received
operation prepared
operation committed
postcondition verified

Those boundaries correspond to meaningful state transitions.

For example:

GENERATING
    ↓
CANDIDATE_READY
    ↓
VERIFYING
    ↓
VERIFIED
    ↓
AWAITING_AUTHORITY
    ↓
AUTHORIZED
    ↓
COMMITTING
    ↓
COMMITTED
    ↓
POSTCONDITION_VERIFIED

A migration in CANDIDATE_READY is usually easier than one halfway through COMMITTING.

This is the same reason databases care about transaction boundaries.

Agent runtimes should too.

4. Make Ownership Transfer Explicit

Step 20 introduced leases and fencing for distributed work.

Handoff must use them.

The dangerous pattern is:

worker A appears dead
worker B resumes
worker A wakes up
both continue

If both workers can mutate external state, the checkpoint system has created a split-brain agent.

A safe transfer looks more like:

worker A owns epoch 41
        ↓
checkpoint sealed
        ↓
lease revoked / expires
        ↓
coordinator advances epoch to 42
        ↓
worker B receives epoch 42
        ↓
mutation gateway rejects epoch 41

The important rule is:

Checkpoint possession does not imply mutation authority.

Only the current fencing epoch should authorize consequential side effects.

5. Never Transfer Authority by Copying Credentials

A portable checkpoint should not contain broad persistent credentials that magically grant the new worker the same privileges.

Instead, authority should be reconstructed from policy.

For example:

checkpoint says:
required authority = A3
approved operation = op-847
approved artifact hash = sha256:...
approval expiry = ...

new worker says:
my placement supports A3
my environment is eligible
my tool contract matches
my current epoch is valid

Then the authority gateway can issue a short-lived scoped capability.

This preserves the Step 29 rule that capability and authority are separate.

Moving the computation does not automatically move permission.

6. Bind Every Checkpoint to a Behavioral Release

A checkpoint created under one behavioral release may not be resumable under another.

Suppose the run was created with:

model-v7
prompt-bundle-18
router-policy-4
tool-contract-9
memory-schema-6
verifier-bundle-12

The replacement worker is running:

model-v8
prompt-bundle-19
tool-contract-10
memory-schema-7

That is not automatically a compatible continuation.

The handoff layer needs a compatibility decision:

@dataclass(frozen=True)
class ResumeCompatibility:
    compatible: bool
    migration_required: bool
    reasons: tuple[str, ...]
    allowed_release_ids: tuple[str, ...]

The safest default for long-running consequential work is often release pinning:

run starts on release R
    ↓
run remains on R until completion

If migration to a new release is necessary, treat it as a behavioral migration—not a worker restart.

7. Preserve Observation Identity

A replacement worker must know what the original worker actually observed.

That means storing evidence identity, not only summaries.

observation
├── source identity
├── retrieval/query parameters
├── timestamp
├── content hash
├── authority/freshness metadata
└── derived interpretation references

Why?

Because otherwise the new worker may silently re-query a live dependency and get different evidence.

That is not continuation.

It is re-observation.

Sometimes re-observation is exactly what we want.

But it must be explicit:

USE_RECORDED_OBSERVATION
REVALIDATE_LIVE_STATE
REFRESH_STALE_OBSERVATION

Those are different operations with different semantics.

8. Separate Facts From Derived Agent State

A useful portable state format distinguishes:

authoritative observations
        ↓
derived structured state
        ↓
agent hypotheses / plans

Do not collapse all three into a prose scratchpad.

For example:

@dataclass(frozen=True)
class PortableRunState:
    observations: tuple[str, ...]
    derived_facts_ref: str
    active_plan_ref: str | None
    candidate_refs: tuple[str, ...]
    unresolved_uncertainty_ref: str | None

If the replacement worker uses a different model, it may reinterpret the same evidence.

That can be legitimate.

But the runtime should still preserve which parts were authoritative input and which parts were model-derived state.

9. Search State Is More Than the Winning Branch

Advanced agents may maintain:

• beam candidates,
• MCTS nodes,
• critic revisions,
• evolutionary populations,
• speculative branches,
• rejected alternatives.

If migration keeps only the current winner, the algorithm has changed.

A portable search checkpoint may need:

node identities
parent-child lineage
visit counts
scores
verification status
pruning reasons
cancellation state
budget consumed
remaining frontier

You do not always need all of this.

But if the scheduler claims the run can resume without changing search semantics, the required search state must be durable.

10. Budget State Must Move Too

A migrated run should not receive a fresh budget accidentally.

That can break both reliability and cost controls.

Preserve:

model spend used
model calls used
search expansions used
tool calls used
browser time used
sandbox time used
verification reserve remaining
wall-clock deadline
retry budget

Otherwise:

worker A spends 80% of budget
        ↓
migration
        ↓
worker B starts at 0%
        ↓
run quietly exceeds policy

The budget ledger should belong to the logical run, not the worker.

11. Verification Evidence Must Follow Exact Artifact Identity

Suppose candidate C1 was verified on worker A.

Worker B loads the checkpoint and normalizes whitespace, regenerates a file, or reserializes structured output.

Is it still C1?

Maybe.

Maybe not.

The verifier evidence should be bound to exact identity:

candidate_hash
workspace_snapshot
verifier_version
verification_inputs
verification_result

If the candidate changes, the old verification should not automatically transfer.

The rule is:

Verified meaning must not be inferred from approximate artifact similarity.

Re-verify when identity changes.

12. Partial Side Effects Require Reconciliation

The hardest handoff occurs after an ambiguous external action.

Example:

POST /deploy
    ↓
timeout
    ↓
worker dies

Did deployment happen?

The replacement worker must not blindly retry.

Instead:

operation_id
    ↓
query authoritative external state
    ↓
COMMITTED / NOT_COMMITTED / UNKNOWN

Then:

COMMITTED
    -> continue with postcondition verification

NOT_COMMITTED
    -> retry if policy allows and idempotency holds

UNKNOWN
    -> preserve UNKNOWN / escalate / reconcile further

This is where idempotency keys and operation ledgers become essential.

Portable execution state must include side-effect lineage, not just reasoning state.

13. Browser Agents Need Special Treatment

Browser state is notoriously difficult to migrate safely.

A browser worker may have:

• cookies,
• CSRF tokens,
• DOM state,
• ephemeral sessions,
• partially completed forms,
• popup windows,
• anti-bot challenges,
• authentication state.

Blindly serializing browser memory is rarely enough.

Useful resumability classes might be:

page can be re-opened from durable URL
session can be reconstructed from scoped credentials
workflow can restart from last confirmed server-side state
workflow cannot be resumed safely

For consequential browser workflows, prefer authoritative server-side state over assumptions about the old DOM.

14. Coding Agents Need Workspace Identity

For coding work, checkpoint portability depends heavily on workspace identity.

A useful handoff may include:

repository URL
base commit SHA
worktree snapshot/hash
uncommitted patch hash
build environment identity
dependency lockfiles
test command definitions
verification outputs

If the repository has changed since the checkpoint:

old patch
    +
new base

is not automatically the same task state.

The runtime may need to:

resume against frozen snapshot
rebase in sandbox
re-run validation
request new approval

A migration that changes the base repository state should be recorded as a state transition, not hidden inside resume logic.

15. Provider Migration Is a Behavioral Change

Moving from one machine to another running the same pinned model may be mostly operational.

Moving from a local model to a frontier API is different.

Even with the same prompt, the behavior changes.

So distinguish:

physical migration
logical migration
behavioral migration

Physical migration

Same behavioral release, compatible runtime, different worker.

Logical migration

Same intended behavior, but state representation requires a compatibility transform.

Behavioral migration

Model, prompt, router, tools, verifier semantics or other behavior-producing components change.

Behavioral migration should invoke Step 24 release rules and Step 30 competence rules.

Do not disguise it as failover.

16. Use a Handoff Protocol

A safe handoff can be explicit:

1. quiesce source worker
2. finish or abort non-checkpointable local operations
3. seal checkpoint
4. persist checkpoint hash
5. record pending side effects
6. release/revoke source ownership
7. select compatible target placement
8. validate release/state/tool/verifier compatibility
9. acquire new lease + fencing epoch
10. revalidate external preconditions
11. restore logical state
12. resume from semantic boundary
13. verify first resumed transition

Notice how much stronger this is than:

pickle.dump(agent)

17. Handoff Should Be Two-Phase

A useful mental model is:

PREPARE HANDOFF
    ↓
checkpoint sealed
compatibility evaluated
target reserved

COMMIT HANDOFF
    ↓
old authority fenced
new owner epoch activated
resume permitted

If target preparation fails, the source worker may continue if still healthy.

If the handoff commits, the old worker must no longer be allowed to mutate.

This resembles transactional ownership transfer because that is effectively what it is.

18. Never Assume “Stopped” Means Stopped

Step 19 already gave us the rule:

cancel requested ≠ underlying work actually stopped

The same is true during migration.

A model request, browser action or tool execution may continue after the orchestration layer thinks the worker is quiesced.

Therefore the handoff record should track:

logical cancellation
physical cancellation status
possible outstanding external work
operation IDs

A new worker should not assume clean ownership merely because the old scheduler marked the task moved.

Fencing remains the ultimate protection for side effects.

19. State Schemas Need Compatibility Rules

Portable state will evolve.

Eventually you will have:

checkpoint schema v3
checkpoint schema v4
checkpoint schema v5

Define directional compatibility.

v5 reader can read v4
v4 reader cannot read v5

Or require explicit migration:

migrate_v4_to_v5(checkpoint)

The migration itself should be deterministic where possible and produce a new artifact with lineage back to the original.

Do not silently mutate old checkpoint records.

20. Some State Should Not Be Portable

Portability is not always desirable.

Examples:

• privileged credentials,
• hardware-bound secrets,
• region-restricted data,
• local security tokens,
• browser sessions prohibited from export,
• sensitive intermediate artifacts,
• provider-specific encrypted state.

A checkpoint should be able to declare:

portable_to_regions = {eu-west}
portable_to_providers = {local}
requires_hardware_attestation = true
contains_non_exportable_state = true

Placement must respect that.

Step 34 chooses eligible targets.

Step 35 now gives it the state constraints required to decide whether migration is actually possible.

21. Handoff Can Reduce Authority

Suppose the source worker was operating in a protected production enclave.

The only available replacement is a less trusted environment.

The correct result might be:

resume observation
resume planning
resume verification
DO NOT resume mutation authority

This is useful.

Handoff does not need to be all-or-nothing.

The target may inherit a reduced authority envelope.

That lets the task continue gathering evidence while waiting for a suitable mutation-capable placement.

22. Handoff Can Trigger Reverification

Migration changes operational context.

Sometimes that alone invalidates previous assumptions.

For example:

• a new region sees a different data replica,
• a new browser session changes server state,
• a new sandbox has different dependencies,
• a new tool version changes semantics,
• the external resource changed while the run was paused.

So a resume policy might say:

if environment identity changed:
    revalidate state

if artifact identity changed:
    reverify artifact

if authority context changed:
    reauthorize

if release changed:
    run compatibility gate

Resume is not merely deserialization.

It is a guarded state transition.

23. Replay and Handoff Should Share the Same State Model

Step 25 introduced replay manifests.

Portable execution should reuse the same provenance model rather than invent a parallel one.

Conceptually:

live run
   ↓
durable state + provenance
   ├── replay later
   └── resume elsewhere now

That is powerful because the same artifacts support:

• failover,
• debugging,
• audit,
• incident reconstruction,
• counterfactual replay,
• regression creation.

Reliability infrastructure compounds when primitives are reused this way.

24. Observe Handoffs as First-Class Events

A trajectory should record events such as:

checkpoint_started
checkpoint_sealed
handoff_requested
source_quiesced
source_fenced
target_selected
compatibility_passed
compatibility_failed
ownership_transferred
state_restored
revalidation_started
revalidation_failed
resume_started
resume_verified
handoff_aborted

Useful fields include:

run_id
checkpoint_id
source_placement
target_placement
source_epoch
target_epoch
release_id
state_schema
reason
compatibility_decision
authority_before
authority_after
resume_boundary

This makes migration visible in Step 26 incident forensics.

25. Measure Handoff Quality

Useful metrics include:

handoff success rate
resume failure rate
checkpoint latency
checkpoint size
recovery time
revalidation rate
handoff-induced UNKNOWN rate
duplicate-side-effect incidents
stale-worker rejection count
state incompatibility rate
handoff rollback rate
authority downgrade rate

Also measure correctness:

verified success after handoff
vs
verified success without handoff

A migration system that reduces infrastructure downtime but increases behavioral failure is not reliable.

26. Benchmark Against Simpler Alternatives

Before building universal live migration, compare it against:

restart from beginning
restart from last deterministic step
retry on same placement
wait for same resource pool
checkpoint only at coarse boundaries
human-assisted recovery

Some workloads are cheap enough that restart is better than complex state portability.

This follows the series-wide rule:

Complexity must earn its cost empirically.

Do not build process migration because distributed-agent architecture diagrams look sophisticated.

Build it when long-running work, expensive evidence gathering, scarce resources or consequential side effects make restart materially worse.

27. Failure Injection Matters

Test handoff under deliberately ugly conditions.

For example:

source dies before checkpoint seal
source dies after seal but before lease release
target dies after acquiring epoch
old worker resumes after fencing
checkpoint blob is missing
checkpoint hash mismatches
state schema is incompatible
verifier bundle unavailable
repository changed during pause
browser session expired
provider changed model alias
external side effect timed out
approval expired during migration
budget expires during handoff
region policy changes mid-run

For every case, ask:

Can duplicate mutation occur?
Can stale evidence be accepted?
Can authority expand accidentally?
Can verification attach to the wrong artifact?
Can budget reset?
Can UNKNOWN be hidden?

Those are more important than whether the worker successfully deserializes JSON.

28. A Minimal Handoff Decision

The first implementation does not need a learned migration controller.

A deterministic compatibility gate is a better starting point.

from dataclasses import dataclass

@dataclass(frozen=True)
class HandoffDecision:
    allowed: bool
    resume_mode: str
    authority_ceiling: str
    requires_revalidation: bool
    requires_reverification: bool
    reasons: tuple[str, ...]


def evaluate_handoff(checkpoint, target):
    reasons = []

    if checkpoint.release_id not in target.supported_release_ids:
        reasons.append("release_incompatible")

    if checkpoint.state_schema_version not in target.readable_state_schemas:
        reasons.append("state_schema_incompatible")

    if not target.can_access(checkpoint.required_data_scope):
        reasons.append("data_scope_incompatible")

    if not target.has_required_verifiers(checkpoint.required_verifiers):
        reasons.append("verifier_unavailable")

    if reasons:
        return HandoffDecision(
            allowed=False,
            resume_mode="NONE",
            authority_ceiling="NONE",
            requires_revalidation=False,
            requires_reverification=False,
            reasons=tuple(reasons),
        )

    return HandoffDecision(
        allowed=True,
        resume_mode="CHECKPOINT",
        authority_ceiling=target.authority_ceiling,
        requires_revalidation=True,
        requires_reverification=False,
        reasons=(),
    )

Later, placement or migration policies can become more adaptive.

But the hard compatibility boundaries should remain deterministic.

29. The Full Architecture

The resulting runtime now looks like this:

logical agent run
      ↓
durable execution state
      ↓
semantic checkpoint
      ↓
handoff coordinator
      ↓
compatibility gate
      ↓
capability-aware placement
      ↓
ownership transfer
      ↓
lease + fencing epoch
      ↓
state restore
      ↓
external-state revalidation
      ↓
resume
      ↓
verification

Around it sit the systems built earlier in the series:

behavioral release manifest
competence envelope
authority policy
SLO / error budget
replay provenance
incident forensics
capability dependency graph
placement policy

The checkpoint is therefore not an isolated runtime feature.

It is another point where the entire reliability architecture comes together.

30. The Real Goal Is Continuity of Meaning

There is a tempting implementation mindset here:

Can we serialize enough state to restart the program elsewhere?

That is the wrong goal.

The stronger question is:

Can the replacement execution continue the same logical task under explicit, compatible evidence, ownership, authority and verification semantics?

Those are not equivalent.

A migrated process can be technically alive while the original task has already lost its meaning.

That is why portable execution state must preserve more than memory.

It must preserve the contract of continuation.

The rule is worth repeating:

A task may move. Its meaning, evidence, and authority must not move implicitly.

What Comes Next?

Once execution state is portable, another problem appears.

A long-running agent may span hours or days.

During that time:

• repository state changes,
• web evidence becomes stale,
• dependencies are updated,
• approvals expire,
• model aliases move,
• policies change,
• other agents modify shared resources,
• users change their intent.

A checkpoint can preserve what the agent knew.

It cannot guarantee that what it knew is still true.

The next stage is therefore temporal consistency and stale-state management:

How does an agent know which assumptions must be refreshed before it continues acting?

That means leases not only on workers, but on facts.

Freshness budgets.

Read-set tracking.

Optimistic concurrency.

Precondition revalidation.

Conflict detection.

And explicit invalidation when the world moves underneath a long-running plan.

Because in a real agent system, the hardest state problem is not merely moving state.

It is knowing when that state has expired.