Audit Agent Replay and Provenance Completeness
Apply Advanced Agents Step 25 to determine whether a historical agent run can actually be reconstructed, explained and compared after models, prompts, tools and state have changed.
How to use this
- Open a repository-aware AI assistant.
- Give it access to the repository or files you want reviewed.
- Copy the prompt below and run it unchanged first.
- Use the evidence it finds to decide what to inspect or change next.
PromptCopy and run against your own project
You are reviewing whether this production agent system supports meaningful deterministic replay and provenance.
Do not treat ordinary logs as sufficient evidence of reproducibility.
Reconstruct the exact evidence required to explain a historical run after the surrounding system has changed.
Inspect the repository and identify how a run records or binds:
- run/trajectory identity,
- behavioral release/version,
- model/provider/model version,
- prompt/template version,
- policy/router version,
- tool schemas and implementation versions,
- retrieved documents and their immutable identities,
- memory inputs,
- external observations,
- environment/configuration snapshot,
- search/branch lineage,
- scheduler/worker attempts,
- side effects,
- verifier version and evidence,
- final outcome.
Build a provenance matrix with columns:
artifact or decision | current identifier | immutable? | persisted? | replayable? | missing dependency
Then select one representative agent run path and determine whether a future investigator could answer:
1. What did the agent actually observe?
2. Which code/prompt/model/policy produced each decision?
3. Which attempts executed versus merely being proposed?
4. Which external state was read?
5. Which side effects occurred?
6. Why was the result accepted or rejected?
7. Which parts can be replayed exactly and which require simulation/substitution?
Look for hidden drift such as:
- mutable prompt names without content hashes,
- model aliases instead of exact versions,
- retrieval IDs that resolve to current rather than historical content,
- memory references without snapshots,
- logs that omit rejected branches or retries,
- verifier PASS/FAIL without verifier evidence/version,
- side effects recorded only in model narration,
- run records that reference mutable configuration,
- timestamps without stable event ordering,
- missing idempotency/attempt identity.
Distinguish:
- deterministic replay,
- evidence replay,
- behavioral reconstruction,
- best-effort reproduction.
Do not claim a stronger property than the repository supports.
Recommend the minimum provenance contract needed to make failures scientifically inspectable. For each missing field, explain which future question becomes impossible without it.
The goal is not perfect token-level determinism. The goal is enough immutable evidence to reconstruct what the system saw, decided, executed and verified.