Audit Whether This Agent Has a Safe and Unambiguous Tool Interface
Apply Agents From First Principles Step 06 to a real repository and inspect whether tool descriptions, schemas, routing and result contracts make the action space understandable and safe.
How to use this
- Open a repository-aware AI assistant.
- Give it access to the repository or files you want reviewed.
- Copy the prompt below and run it unchanged first.
- Use the evidence it finds to decide what to inspect or change next.
PromptCopy and run against your own project
You are auditing the tool boundary of an AI agent.
Your job is to determine whether the model is being given a clear, typed and safe action space or is being forced to guess between overlapping tools and ambiguous schemas.
Do not begin by changing prompts or model choice.
First reconstruct the tool surface from code.
Work in this order:
1. Inventory every model-callable tool
For each tool record:
- name,
- description,
- parameter schema,
- required vs optional fields,
- return schema,
- side effects,
- authority level,
- error modes,
- retry/idempotency characteristics.
2. Audit semantic overlap
- Identify tools whose descriptions or capabilities overlap.
- Identify vague verbs such as search, lookup, fetch, run, execute, update, process.
- Check whether the model can distinguish discovery vs exact lookup, read vs write, preview vs commit, local vs remote, query vs mutation.
- Find parameters whose meaning depends on undocumented conventions.
3. Audit schemas
- Check whether enums could replace free text.
- Check whether mutually exclusive arguments are represented explicitly.
- Identify oversized schemas where most fields are irrelevant to most calls.
- Identify nullable/default behavior that can cause unintended actions.
- Check validation before execution.
4. Audit routing
- Find where a proposed action is mapped to a tool.
- Determine whether routing is model-only, deterministic, constrained, or post-validated.
- Check whether tool availability is scoped by task, state, competence, or authority.
- Identify fallback behavior after invalid calls.
5. Audit results
- Determine whether tool results are typed enough for the agent to distinguish success, partial success, no result and failure.
- Check whether prose result text can be misread as authoritative state.
- Identify missing IDs, version markers, timestamps, status codes, changed-state evidence or provenance.
6. Audit side-effect safety
- Separate read tools from mutation tools.
- Identify destructive or externally visible tools.
- Check confirmation/authority gates, idempotency keys and postconditions.
- Determine whether retries can duplicate writes.
7. Adversarial cases
Reason through:
- two tools both plausibly match the same request,
- required semantic distinction is only in a long description,
- model omits an optional field whose default is dangerous,
- tool returns partial success,
- malformed result still looks like success,
- retry repeats an external mutation.
Output:
A. Tool inventory
B. Ambiguity/overlap matrix
C. Schema weaknesses
D. Routing weaknesses
E. Result-contract weaknesses
F. Side-effect and retry risks
G. Smallest interface corrections
H. Concrete routing/schema tests
Classify each finding as:
- CONFIRMED FROM CODE
- STRONGLY INFERRED
- NEEDS RUNTIME EVIDENCE
Treat tool design as action-space design.
A stronger model does not fix an ambiguous action space.