Use with AI · Chapter Application

Audit Whether This Agent Search Strategy Is Worth Its Extra Compute

Apply Agents From First Principles Step 08 to a real repository and inspect whether branching, scoring, pruning and search budgets improve outcomes over simpler one-path execution.

Chapter Application Agents From First Principles Agents using Best-of-N, beam search, tree search or multi-trajectory exploration Advanced
Read the source chapter → View the learning path →

How to use this

  1. Open a repository-aware AI assistant.
  2. Give it access to the repository or files you want reviewed.
  3. Copy the prompt below and run it unchanged first.
  4. Use the evidence it finds to decide what to inspect or change next.
PromptCopy and run against your own project
You are auditing an AI agent that explores more than one candidate action, plan or trajectory.

Your job is to determine whether search is implemented as a disciplined decision process and whether it earns its latency and compute cost.

Do not assume more branches mean better reasoning.
Do not assume a model-generated score is an objective measure of branch quality.

Work in this order:

1. Reconstruct the search topology
- Identify where branching occurs: candidate answers, plans, actions, tool paths or full trajectories.
- Record branching factor, depth, beam width, retry count and maximum live states.
- Identify whether states are copied, shared or mutated across branches.

2. Identify the state representation
For every search node determine what is preserved:
- task state,
- observations,
- tool results,
- files/external side effects,
- model context,
- accumulated cost,
- score,
- parent/lineage,
- verification status.

3. Audit candidate generation
- Determine whether branches are genuinely diverse or near-duplicates.
- Check temperature/prompt/tool/state differences that create diversity.
- Identify deterministic bottlenecks that make multiple branches equivalent.

4. Audit scoring and pruning
- Find every scorer, critic, heuristic or verifier used to rank branches.
- Determine what evidence each score consumes.
- Check for self-scoring where the same model generates and judges without external signal.
- Identify premature pruning that can eliminate slower but better trajectories.
- Check tie-breaking and score normalization.

5. Audit side effects and isolation
- Determine whether speculative branches can mutate shared files, databases, browser state or remote systems.
- Check whether branches use sandboxes/worktrees/snapshots or another isolation mechanism.
- Identify side effects that cannot be rolled back when a branch loses.

6. Audit budgets
Measure or reconstruct:
- model calls,
- tokens,
- wall-clock time,
- tool calls,
- external requests,
- memory/state growth.

Check whether the budget is global and whether nested search can bypass it.

7. Compare against simpler baselines
Require an equal-budget comparison against at least:
- single-path execution,
- Best-of-N at the final answer only,
- shallower/narrower search where appropriate.

Compare task success, verified correctness, latency and cost.

8. Failure injection
Reason through:
- all branches repeat the same early mistake,
- scorer prefers polished but incorrect trajectories,
- losing branch already produced an irreversible side effect,
- branch state leaks into another branch,
- best branch is pruned early,
- search budget explodes on one difficult task.

Output:

A. Search topology
B. Search-state contract
C. Generation-diversity assessment
D. Scoring/pruning audit
E. Isolation and side-effect risks
F. Budget model
G. Equal-budget baseline plan
H. Smallest corrections
I. Tests/benchmarks required to justify keeping search

Classify every finding as:
- CONFIRMED FROM CODE
- STRONGLY INFERRED
- NEEDS RUNTIME EVIDENCE

The standard is not 'search is more sophisticated.'
The standard is: under a comparable budget, search produces measurably better verified outcomes often enough to justify its complexity.