Agents From First Principles 02: AI Agent Gives Inconsistent Answers? Generate Multiple Candidates and Rank Them
One of the first things you notice when you build anything around a large language model is that the same prompt does not always produce the same quality of answer.
Sometimes the first response is excellent.
Sometimes it is merely acceptable.
Sometimes it misses the point entirely.
That creates a very common agent-engineering question:
If the model is inconsistent, should the agent trust the first answer it gets?
Often, no.
A simple and surprisingly powerful alternative is:
prompt
↓
generate several candidates
↓
score or compare them
↓
select the best
This is usually called Best-of-N.
It is one of the simplest ways to improve an agent without changing the underlying model.
The model stays the same.
The agent changes the computation around it.
That distinction matters.
This post builds Best-of-N from first principles, shows why it works, explains where it fails, and gives you a practical debugging framework for questions like:
- Why does my AI agent give different answers every time?
- How do I rank multiple LLM outputs?
- Should I use an LLM judge or a learned scorer?
- Why does Best-of-N sometimes pick the wrong answer?
- Why are all my candidates nearly identical?
- Does generating more samples actually improve quality?
- How many candidates should an agent generate?
- How do I measure whether the extra model calls are worth it?
The key idea is simple:
If one sample is unreliable, create a small search over candidate answers and make selection an explicit part of the system.
But that only helps if the selector is reliable enough to distinguish good candidates from bad ones.
So there are really two problems:
1. Generate useful diversity.
2. Select well.
Everything else follows from those two requirements.
1. The one-shot baseline
Start with the simplest possible system:
def answer(prompt, llm):
return llm(prompt)
Conceptually:
prompt
↓
model
↓
answer
There is nothing wrong with this.
In fact, this should usually be your baseline.
If one model call already solves the task reliably, adding candidates, judges and ranking logic only adds:
- latency,
- cost,
- failure modes,
- implementation complexity.
The purpose of Best-of-N is not to make the system look more agentic.
The purpose is to fix a measurable failure:
the quality of one-shot generation is too variable.
That means the first thing to measure is not Best-of-N performance.
Measure the one-shot baseline.
For a test set, record something like:
task_id
candidate
verified_success
latency_ms
model_calls
If one-shot success is already 98%, a four-candidate agent may be a terrible trade.
If one-shot success is 61% and the model often produces one good answer among several attempts, Best-of-N becomes much more interesting.
2. Generate more than one candidate
The smallest Best-of-N generator looks like this:
def generate_candidates(prompt, llm, n=4):
return [llm(prompt) for _ in range(n)]
Now the computation is:
┌→ candidate 1
├→ candidate 2
prompt → model├→ candidate 3
└→ candidate 4
This gives us options.
But options alone do not improve anything.
We still need to choose.
So we add a scorer:
def choose_best(candidates, score_fn):
scored = [
(score_fn(candidate), candidate)
for candidate in candidates
]
return max(scored, key=lambda item: item[0])
The full system becomes:
prompt
↓
generate N candidates
↓
score each candidate
↓
select highest score
↓
final answer
That is already an agent technique.
We have turned one model call into a small search-and-selection procedure.
3. Why this can work
Suppose a model has some probability of producing a good candidate on any single attempt.
The exact outputs are stochastic, so repeated samples can explore different solutions.
If the model occasionally produces a strong answer but not consistently, multiple generation gives the agent more chances to find one.
The intuition is:
one sample
↓
one chance
multiple samples
↓
several chances
But there is a catch.
Generating a good answer is not enough.
The agent must recognize it.
That is why Best-of-N is really:
search quality
×
selection quality
A perfect generator with a terrible selector can still fail.
A perfect selector cannot help if all candidates are bad.
So debug them separately.
4. The two failure surfaces
When a Best-of-N agent performs badly, ask two separate questions.
Question A: Was there a good candidate in the set?
If no, generation is the problem.
Question B: If there was a good candidate, did the selector choose it?
If no, ranking is the problem.
This distinction is enormously useful.
Imagine the candidates are:
A: wrong
B: correct
C: incomplete
D: plausible but wrong
If the system returns D, the model actually succeeded at generation.
The selector failed.
If all four are wrong, the selector never had a chance.
That gives us two metrics:
oracle@N
selector_accuracy_given_oracle
You can define them informally as:
oracle@N:
Did at least one candidate solve the task?
selector_accuracy_given_oracle:
When a correct candidate existed, did we choose it?
These two metrics tell you much more than final success rate alone.
5. A tiny Best-of-N agent
Here is a complete minimal implementation:
from dataclasses import dataclass
from typing import Callable
@dataclass
class Candidate:
text: str
score: float | None = None
class BestOfNAgent:
def __init__(
self,
llm: Callable[[str], str],
scorer: Callable[[str, str], float],
n: int = 4,
):
self.llm = llm
self.scorer = scorer
self.n = n
def run(self, prompt: str) -> Candidate:
candidates = [
Candidate(self.llm(prompt))
for _ in range(self.n)
]
for candidate in candidates:
candidate.score = self.scorer(
prompt,
candidate.text,
)
return max(
candidates,
key=lambda c: c.score,
)
Notice how little machinery we need.
There is no planner.
No memory.
No tree search.
No multi-agent coordination.
Just:
generate
↓
evaluate
↓
select
That is why Best-of-N belongs early in an agents curriculum.
It introduces the idea that an agent can spend additional inference compute to search over alternatives.
6. What should score the candidates?
This is the most important design choice.
There are several broad options.
Option 1: deterministic verification
If the task has an objective checker, use it.
Examples:
- unit tests,
- compiler success,
- exact numerical answer,
- schema validation,
- database constraint,
- benchmark metric,
- file existence,
- expected API response.
Conceptually:
candidate
↓
external verifier
↓
objective score
This is usually stronger than asking another LLM whether the answer looks correct.
For example:
def score_code(candidate: str) -> float:
result = run_tests(candidate)
return result.passed / result.total
Whenever the environment can measure success directly, let the environment judge.
Option 2: heuristic scoring
Sometimes a simple deterministic heuristic is enough.
For example:
def score(candidate: str) -> float:
score = 0.0
if "TODO" not in candidate:
score += 1.0
if len(candidate) < 2000:
score += 0.5
if "```python" in candidate:
score += 0.5
return score
This is crude, but sometimes the target property really is simple.
Do not automatically introduce another model when ordinary code can evaluate the criterion.
Option 3: learned scorer
This is where the previous Models From First Principles series becomes directly useful.
A scorer can be something like:
prompt + candidate
↓
quality model
↓
score
For example:
MR.Q
EBT
SICQL
HRM
The agent does not need to know how the scorer was built.
It only needs a stable interface:
score = scorer(prompt, candidate)
That is a powerful architectural boundary.
The model series explained what these scorers compute.
The agents series now shows how a system can use them.
Option 4: LLM judge
Another common pattern is:
candidate A
candidate B
↓
LLM judge
↓
preferred candidate
For example:
def judge_pair(prompt, a, b, llm):
judge_prompt = f"""
Task:
{prompt}
Candidate A:
{a}
Candidate B:
{b}
Choose the better candidate.
Return exactly A or B.
"""
result = llm(judge_prompt).strip()
return a if result == "A" else b
This is easy to implement.
It is also easy to overtrust.
An LLM judge is still a model.
It can have:
- positional bias,
- verbosity bias,
- style bias,
- self-preference,
- rubric drift,
- inconsistent decisions.
So treat judge quality as something to evaluate, not assume.
7. Pairwise ranking
If your scorer naturally compares two outputs rather than assigning absolute scores, use pairwise ranking.
Suppose we have:
A
B
C
D
A simple tournament can be:
A vs B → B
B vs C → C
C vs D → C
winner = C
Code:
def tournament(candidates, prefer):
best = candidates[0]
for challenger in candidates[1:]:
best = prefer(best, challenger)
return best
This is similar to what many practical generation-and-selection agents do.
But be careful.
The winner can depend on comparison order.
For example:
A beats B
B beats C
C beats A
Preferences do not have to be transitive.
That means tournament order is another thing you may need to test.
8. Position bias in LLM judging
One common problem is that the judge prefers the first or second candidate for reasons unrelated to quality.
A simple diagnostic is to swap positions.
Run:
A vs B
Then:
B vs A
If the result changes frequently, your judge is position-sensitive.
You can record:
forward_winner
reverse_winner
consistent?
Example:
def symmetric_pairwise_judge(a, b, judge):
first = judge(a, b)
second = judge(b, a)
if first == a and second == a:
return a
if first == b and second == b:
return b
return None
This costs more, but it makes judge instability visible.
Do not hide disagreement.
Disagreement is useful telemetry.
9. Score collapse
A learned scorer may assign nearly identical values to every candidate.
For example:
A 0.812
B 0.814
C 0.811
D 0.813
Technically the system can choose B.
But the margin is tiny.
That raises a question:
Does the scorer actually have enough resolution to support this decision?
Track the score spread:
spread = max(scores) - min(scores)
And top-two margin:
sorted_scores = sorted(scores, reverse=True)
margin = sorted_scores[0] - sorted_scores[1]
A tiny margin should not automatically be interpreted as strong preference.
Possible strategies include:
- return the top candidate but log low confidence,
- use a stronger judge only when the margin is small,
- ask an external verifier,
- generate additional candidates,
- abstain.
This naturally leads toward adaptive systems later.
10. Your candidates may not actually be diverse
Generating four answers does not guarantee four meaningfully different answers.
You may get:
Candidate A:
Use a dictionary cache...
Candidate B:
A dictionary-based cache can...
Candidate C:
You should cache values in a dictionary...
Candidate D:
The best solution is a dictionary cache...
That is four generations but effectively one idea.
Best-of-N gains little when samples are strongly correlated.
A useful metric is candidate similarity.
For embeddings:
similarity = cosine(candidate_a, candidate_b)
Or use simpler lexical measures if that is enough.
Track something like:
mean_pairwise_similarity
max_pairwise_similarity
unique_candidate_rate
If all candidates are almost identical, the problem may be generation diversity rather than ranking.
11. Increasing temperature is not automatically the answer
A natural reaction is:
My candidates are too similar. Increase temperature.
Maybe.
But higher sampling randomness can also produce more low-quality candidates.
The real trade-off is:
diversity
vs
candidate quality
So experiment.
For each sampling configuration, record:
oracle@N
final_success
candidate_similarity
invalid_output_rate
You may discover that a modest temperature gives enough diversity without destroying baseline quality.
Or you may discover that changing the prompt produces more useful variation than changing sampling parameters.
12. Prompt for different approaches
Instead of asking the same question four times, explicitly ask for different strategies.
For example:
strategies = [
"Solve directly and minimally.",
"Look for edge cases first.",
"Try a data-structure-oriented solution.",
"Try a correctness-first solution.",
]
Then:
def generate_diverse(prompt, llm):
return [
llm(f"{prompt}\n\nApproach: {strategy}")
for strategy in strategies
]
Now diversity is partly structural rather than purely stochastic.
Conceptually:
same goal
├→ strategy A
├→ strategy B
├→ strategy C
└→ strategy D
That often gives a selector more meaningful alternatives.
13. Best-of-N can amplify judge mistakes
This is an important failure mode.
Suppose one-shot generation returns a reasonable answer.
Then you generate eight candidates.
Among them, one candidate is extremely polished but subtly wrong.
If the judge prefers style over correctness, increasing N may make failure more likely because the larger search gives the judge more opportunities to select a persuasive mistake.
So Best-of-N is not guaranteed to improve monotonically with N.
You should actually test:
N=1
N=2
N=4
N=8
N=16
Measure final verified success.
You may get:
N=1 68%
N=2 74%
N=4 79%
N=8 78%
N=16 76%
That is entirely plausible.
More search only helps if selection quality keeps up.
14. How many candidates should you generate?
There is no universal answer.
The right N depends on:
- one-shot reliability,
- candidate diversity,
- selector reliability,
- model latency,
- model cost,
- task value,
- whether generation can run in parallel.
Treat N as a system parameter.
Benchmark it.
A simple table might look like:
| N | Success | Mean Latency | Model Calls | Cost / Success |
|---|---|---|---|---|
| 1 | 0.68 | 1.2s | 1 | 1.00× |
| 2 | 0.74 | 1.3s parallel | 2 | 1.84× |
| 4 | 0.79 | 1.5s parallel | 4 | 3.44× |
| 8 | 0.78 | 2.1s parallel | 8 | 6.97× |
The best quality configuration may not be the best production configuration.
15. Parallel generation changes the latency trade-off
If candidate calls are independent, generate them concurrently.
Conceptually:
┌→ model call 1 ─┐
├→ model call 2 ─┤
prompt ──────┼→ model call 3 ─┼→ ranking
└→ model call 4 ─┘
In asynchronous Python:
import asyncio
async def generate_parallel(prompt, call_llm, n=4):
tasks = [
call_llm(prompt)
for _ in range(n)
]
return await asyncio.gather(*tasks)
This can reduce wall-clock latency substantially compared with sequential sampling.
But model-call count and compute cost still increase.
Parallel does not mean free.
16. Deduplicate before expensive judging
If candidate evaluation is expensive, remove near-duplicates first.
A simple textual version might be:
def deduplicate(candidates):
seen = set()
unique = []
for candidate in candidates:
normalized = " ".join(candidate.lower().split())
if normalized in seen:
continue
seen.add(normalized)
unique.append(candidate)
return unique
A semantic version can use embeddings.
Pipeline:
generate 8
↓
deduplicate
↓
5 meaningfully distinct
↓
expensive scorer
This is often a better use of inference budget.
17. Ranking with multiple signals
Sometimes no single evaluator captures what you care about.
For code, you might care about:
correctness
simplicity
runtime
style
Then combine explicit signals:
score = (
0.60 * test_score
+ 0.20 * simplicity_score
+ 0.10 * runtime_score
+ 0.10 * style_score
)
The exact weights are task-dependent.
The important architectural lesson is that selection can be decomposed too.
candidate
├→ verifier
├→ scorer
├→ cost model
└→ heuristic
↓
aggregate decision
This is already moving toward richer decision systems.
18. Rank first, verify second
For expensive verification, use a cascade.
Example:
8 generated candidates
↓
cheap scorer
↓
top 2
↓
expensive verifier
↓
final
This is often better than applying the most expensive evaluator to every candidate.
The pattern is:
cheap broad filter
↓
expensive narrow verification
This idea will reappear throughout agent design.
19. Verify first, rank second
Sometimes correctness is binary and cheap to test.
Then invert the cascade:
candidates
↓
objective validation
↓
valid candidates only
↓
rank for quality/style/cost
For example:
valid = [
c for c in candidates
if tests_pass(c)
]
Then choose among valid candidates.
This is often much safer than allowing a learned judge to trade correctness against presentation.
20. Common failure: the judge is judging the wrong thing
Suppose your product needs concise answers.
But your judge prompt says:
Choose the most complete and detailed response.
The selector will systematically prefer verbosity.
That is not a model problem.
It is an objective problem.
The ranking objective must correspond to the real decision.
Ask:
What property are we actually trying to optimize?
Examples:
correctness
user preference
latency
safety
clarity
profit
conversion
ranking quality
Do not use a generic “quality” score when your real task has a specific objective.
21. Common failure: using the same model as generator and judge
This can be useful.
It can also create correlated errors.
If the generator has a systematic misconception, the same model acting as judge may share it.
That does not mean you should never use the same model.
It means you should measure the consequence.
Compare:
generator A + judge A
generator A + judge B
generator A + learned scorer
generator A + external verifier
The important question is not architectural purity.
It is whether the selector catches the generator’s failure modes.
22. Common failure: candidate order leaks into the decision
If you score candidates sequentially and maintain a current winner, order may matter.
Example:
best = candidates[0]
for candidate in candidates[1:]:
best = compare(best, candidate)
This is efficient.
But if comparison is noisy or non-transitive, shuffling the candidate list can change the winner.
Test it.
import random
def order_stability(candidates, tournament, trials=20):
winners = []
for _ in range(trials):
shuffled = candidates[:]
random.shuffle(shuffled)
winners.append(tournament(shuffled))
return winners
If winners vary dramatically, your selector is unstable.
That instability should be visible.
23. Common failure: the candidates differ only cosmetically
Suppose the scorer sees:
A: concise correct answer
B: same answer with headings
C: same answer with bullets
D: same answer with a summary
A Best-of-N system may appear sophisticated while exploring almost no semantic space.
So inspect candidate diversity manually during development.
Log snippets.
Cluster embeddings.
Compare solution strategies.
Ask:
Are these genuinely different attempts?
If not, generating more of them will not rescue the system.
24. Common failure: using Best-of-N where deterministic verification would solve the problem
Suppose an agent generates SQL.
You could:
generate 8 queries
judge them with another LLM
But perhaps you can safely execute candidate queries against a test database and check the result.
Then use the environment.
Likewise for code:
run tests
For mathematics:
substitute result
For structured output:
validate schema
Agent systems are strongest when they exploit objective signals where available.
25. Common failure: hidden cost explosion
A one-shot agent may make one model call.
A Best-of-8 system may make:
8 generation calls
+ 7 tournament judge calls
= 15 model calls
If you then add reversed pairwise judging:
8 generation
+ 14 judge
= 22 calls
Quality may improve.
But the economics changed completely.
Track:
model_calls_per_task
input_tokens
output_tokens
wall_clock_latency
verified_success
cost_per_success
Do not measure only answer quality.
26. Best-of-N as search
It is useful to think of this as the smallest search algorithm in the series.
Search space:
possible model outputs
Search procedure:
sample N outputs
Evaluation function:
scorer / judge / verifier
Selection:
argmax score
So Best-of-N is essentially:
candidates = sample(policy, n=N)
values = evaluate(candidates)
return candidates[argmax(values)]
This framing is important because later techniques extend exactly these pieces.
Tree search adds branching.
Evolution adds mutation and selection over generations.
Adaptive agents change how much search to perform.
But the basic idea starts here:
generate alternatives, evaluate them, keep something better.
27. A stronger implementation
Here is a more practical version that keeps telemetry.
from dataclasses import dataclass, field
from typing import Callable
@dataclass
class Candidate:
text: str
score: float | None = None
metadata: dict = field(default_factory=dict)
@dataclass
class BestOfNResult:
winner: Candidate
candidates: list[Candidate]
score_spread: float
top_margin: float
class BestOfNAgent:
def __init__(
self,
generate: Callable[[str], str],
score: Callable[[str, str], float],
n: int = 4,
):
if n < 1:
raise ValueError("n must be >= 1")
self.generate = generate
self.score = score
self.n = n
def run(self, prompt: str) -> BestOfNResult:
candidates = [
Candidate(text=self.generate(prompt))
for _ in range(self.n)
]
for candidate in candidates:
candidate.score = float(
self.score(prompt, candidate.text)
)
ranked = sorted(
candidates,
key=lambda c: c.score,
reverse=True,
)
scores = [c.score for c in ranked]
spread = max(scores) - min(scores)
margin = (
scores[0] - scores[1]
if len(scores) > 1
else float("inf")
)
return BestOfNResult(
winner=ranked[0],
candidates=ranked,
score_spread=spread,
top_margin=margin,
)
Now the result contains not just the chosen answer, but information about the decision.
That is useful for debugging.
28. Add an abstention threshold
If the selector cannot confidently distinguish candidates, do not pretend it can.
For example:
if result.top_margin < 0.01:
return "LOW_CONFIDENCE_SELECTION"
Or escalate:
small score margin
↓
stronger verifier
This is our first glimpse of adaptive computation.
The system spends extra effort only when the cheap decision looks uncertain.
29. Best-of-N with deterministic validation
A robust pattern is:
def choose_candidate(candidates):
valid = [
candidate
for candidate in candidates
if validate(candidate)
]
if not valid:
return None
return max(
valid,
key=quality_score,
)
Conceptually:
N candidates
↓
validation
↓
valid subset
↓
quality ranking
↓
winner
This makes correctness a gate instead of a soft preference.
That distinction can be extremely important.
30. Evaluate the selector independently
Do not only evaluate the end-to-end system.
Create labelled candidate sets where you already know which answer should win.
Then test the selector alone.
Example dataset:
cases = [
{
"prompt": "...",
"candidate_a": "...",
"candidate_b": "...",
"preferred": "a",
},
]
Then compute pairwise accuracy.
This answers:
Is the selector actually capable of making the decisions we are asking it to make?
That is much more informative than treating ranking as an invisible implementation detail.
31. Evaluate generation independently
Likewise, measure whether additional candidates increase the chance that a correct answer appears.
For each N:
oracle@1
oracle@2
oracle@4
oracle@8
Suppose:
oracle@1 = 0.66
oracle@2 = 0.75
oracle@4 = 0.86
oracle@8 = 0.88
Then most generation benefit arrives by N=4.
If final success at N=4 is only 0.72, your selector is leaving substantial quality on the table.
That immediately tells you where to work next.
32. Measure selection regret
If you have an objective reward, measure how much value the selector loses compared with the best candidate generated.
For a candidate set:
best available reward = 0.95
selected reward = 0.74
regret = 0.21
Formally:
regret = oracle_reward - selected_reward
Low regret means selection is working well.
High regret means the search is finding value that the selector fails to capture.
This is a very useful agent diagnostic.
33. Best-of-N versus retry-on-failure
These are not the same technique.
Retry-on-failure:
generate
↓
validate
↓
failed?
├─ no → done
└─ yes → generate again
Best-of-N:
generate several
↓
compare
↓
choose
Retry is conditional on failure.
Best-of-N deliberately creates competition even when the first answer might be valid.
Which one you need depends on the failure.
If outputs are mostly correct but occasionally invalid, retry may be cheaper.
If several valid outputs exist but quality varies, Best-of-N may be more useful.
34. Best-of-N versus self-consistency
These ideas are related but not identical.
Best-of-N generally means:
multiple candidates
↓
external ranking or scoring
↓
select one
Self-consistency often means:
multiple reasoning paths
↓
extract final answers
↓
aggregate / majority vote
For example:
42
42
41
42
39
Majority vote gives 42.
No explicit quality model is required.
Self-consistency becomes more relevant when outputs can be reduced to a comparable final answer.
We will revisit richer reasoning-path techniques in the advanced agents series.
35. Best-of-N versus mixture of experts
Best-of-N can use the same generator repeatedly:
model A
model A
model A
model A
Mixture-of-experts-style systems can generate candidates from different experts:
model A
model B
code specialist
retrieval specialist
Both create alternatives.
But the source of diversity differs.
Best-of-N explores stochastic variation from one policy.
Mixture systems explore variation across specialized policies.
We will keep the advanced routing version for the later series.
36. Common debugging checklist
If you found this article because your agent gives inconsistent answers, debug in this order.
1. Measure one-shot variance
Run the same task multiple times.
Is quality actually unstable?
2. Check oracle@N
When you generate more candidates, does the chance of getting a good one increase?
If no, ranking is not your problem.
3. Inspect candidate diversity
Are you getting different solutions or paraphrases of the same solution?
4. Test the selector independently
Can it reliably choose labelled preferred candidates?
5. Swap candidate order
Does the judge change its mind?
6. Measure top score margin
Is the selector making confident distinctions or arbitrary tie-breaks?
7. Compare objective verification where possible
Can tests or environment signals replace subjective judging?
8. Sweep N
Does N=8 actually outperform N=4?
9. Track model-call cost
What is the cost per successful task?
10. Keep N=1 as the baseline
Never lose the simple system you are trying to beat.
37. A useful experiment matrix
Run something like:
| Generator | N | Selector | Final Success | Oracle@N | Selector Regret | Calls |
|---|---|---|---|---|---|---|
| Model A | 1 | none | … | … | … | 1 |
| Model A | 2 | learned scorer | … | … | … | 4 |
| Model A | 4 | learned scorer | … | … | … | 8 |
| Model A | 4 | LLM judge | … | … | … | … |
| Model A | 4 | verifier | … | … | … | … |
This makes the architecture testable.
Without this, it is very easy to convince yourself that “the agent seems better” simply because the pipeline is more elaborate.
38. Do you actually need Best-of-N?
Use this decision rule.
Is one-shot quality reliable enough?
yes
↓
stop
no
↓
Does repeated sampling sometimes produce a much better answer?
no
↓
Fix the generator/model/prompt instead
yes
↓
Can you rank or verify candidates reliably?
no
↓
Build a better evaluator first
yes
↓
Try Best-of-N
Best-of-N is appropriate when:
- candidate quality varies,
- useful alternatives appear across samples,
- selection can be made reliably,
- extra inference cost is acceptable.
It is a poor choice when:
- all samples fail in the same way,
- outputs are almost identical,
- the selector is weaker than the generator,
- deterministic verification would solve the problem more directly,
- latency constraints are strict,
- the task is already reliably solved in one call.
39. What this adds to our agent
In the previous post, the agent learned to produce structured actions safely.
Now we have added another mechanism:
single proposal
↓
multiple proposals
↓
evaluation
↓
selection
Our agent can now search a small set of alternatives before committing.
That is a significant conceptual step.
But it still does not improve an answer after generation.
It only chooses among generated answers.
The next question is obvious:
What if the first candidate is close, but has a specific flaw we can identify and repair?
That leads to the next technique:
generate
↓
critique
↓
revise
Next: AI Agent Keeps Making the Same Mistake? Add a Critique-and-Revision Loop
In the next post we will move from selection to improvement.
Instead of generating several independent answers and choosing one, the agent will inspect its own current candidate, identify a concrete defect, and produce a revised version.
That gives us another increasingly useful agent primitive:
draft
↓
critique
↓
revision
↓
verify
And once again, the key question will not be whether reflection sounds intelligent.
It will be:
Does critique-and-revision actually fix failures that one-shot generation and Best-of-N do not?
