A reasoning agent gives you one answer.

You run it again.

It gives you another.

You change nothing important:

• same task,
• same tools,
• same model family,
• same broad context.

Yet the result changes.

That is not necessarily a bug.

A probabilistic model is allowed to produce more than one plausible trajectory.

The engineering question is different:

How should an agent system use that variation?

One common answer is self-consistency.

Generate several independent reasoning trajectories.

Normalize their outcomes.

Measure agreement.

Then use that agreement as one signal when deciding what to do next.

The important phrase is one signal.

Consensus is not truth.

Five agents can agree on the same wrong answer.

Ten samples from the same model can repeat the same bad assumption.

A majority vote can make a systematic error look more convincing.

So this post is not about asking the model the same question five times and blindly trusting the most common answer.

It is about building a runtime that can distinguish:

agreement
    from
correctness

and then use disagreement intelligently.

The searchable problem

People building reasoning agents tend to encounter variations of the same questions:

• Why does my AI agent give a different answer every time?
• How do I make an LLM agent more consistent?
• Should I run the same prompt multiple times?
• How many reasoning samples should I generate?
• Does majority voting improve LLM accuracy?
• What is self-consistency in agent systems?
• Why do multiple agents agree on the wrong answer?
• How do I detect correlated reasoning failures?
• How should an agent react when candidate answers disagree?
• Should disagreement trigger another model call or external verification?

Self-consistency is useful because it turns output variance into observable runtime state.

Instead of pretending the first answer is definitive, the runtime can ask:

How many plausible trajectories exist?

Do they converge?

Where do they diverge?

Is the disagreement resolvable from the environment?

That is a much more useful framing than simply trying to suppress stochasticity.

1. Start with the simplest possible baseline

Suppose our model is represented by a function:

def reason(task: str) -> str:
    ...

A one-shot system does this:

answer = reason(task)
return answer

That is the baseline.

Before adding self-consistency, keep it.

You need to know whether multiple samples actually improve anything.

If one deterministic call already solves the task reliably, sampling eight trajectories is just added cost.

2. The smallest self-consistency loop

The simplest implementation is:

def sample_answers(reason, task: str, n: int) -> list[str]:
    return [reason(task) for _ in range(n)]

Then count equivalent answers.

For a toy classification problem:

from collections import Counter

answers = [
    "A",
    "B",
    "A",
    "A",
    "B",
]

counts = Counter(answers)

print(counts)

Result:

A: 3
B: 2

A naive self-consistency system would return A.

But this is only safe when the output space is already canonical.

Real reasoning outputs rarely are.

3. Free-form answers need normalization

These answers may all mean the same thing:

42

The answer is 42.

I calculate the result as forty-two.

final_answer: 42

A string-frequency vote would incorrectly treat them as different.

So self-consistency normally needs an explicit answer extraction layer.

For example:

from dataclasses import dataclass


@dataclass(frozen=True)
class ReasoningResult:
    raw_output: str
    final_answer: str

The model may generate rich intermediate computation, but the runtime compares the canonical outcome.

For structured tasks, prefer structured output:

{
  "answer": "restart_service",
  "confidence": 0.71
}

or:

{
  "root_cause": "expired_database_credentials",
  "recommended_action": "rotate_credentials"
}

The less ambiguity there is in the final representation, the more meaningful agreement becomes.

4. Never let the model define equivalence implicitly

A subtle failure appears when the runtime asks another LLM:

Are these answers equivalent?

Sometimes that is necessary.

But now the consensus mechanism contains another probabilistic model.

You have moved the uncertainty rather than removed it.

Prefer deterministic normalization whenever the domain permits it.

Examples:

classification      → enum
numeric answer      → parsed number
code repair         → test result / patch hash
tool selection      → tool ID
SQL task            → normalized result set
support routing     → queue ID
incident diagnosis  → canonical cause label

Use semantic equivalence only when the answer space genuinely cannot be reduced to a stable representation.

5. Agreement is a measurement

Suppose we generate eight samples:

A A A A A B B C

The top answer has five votes.

A useful runtime does not merely store:

winner = A

It stores the distribution.

from dataclasses import dataclass


@dataclass(frozen=True)
class ConsensusStats:
    winner: str
    winner_votes: int
    total_votes: int
    agreement: float
    unique_answers: int
    margin: float

For the example:

winner        = A
winner_votes  = 5
total_votes   = 8
agreement     = 0.625
unique_answers = 3

The top-two margin is:

(5 - 2) / 8 = 0.375

Now disagreement becomes inspectable.

6. A simple consensus function

from collections import Counter


def consensus(answers: list[str]) -> ConsensusStats:
    if not answers:
        raise ValueError("answers must not be empty")

    counts = Counter(answers)
    ranked = counts.most_common()

    winner, winner_votes = ranked[0]
    second_votes = ranked[1][1] if len(ranked) > 1 else 0

    total = len(answers)

    return ConsensusStats(
        winner=winner,
        winner_votes=winner_votes,
        total_votes=total,
        agreement=winner_votes / total,
        unique_answers=len(counts),
        margin=(winner_votes - second_votes) / total,
    )

This gives the runtime more information than a majority vote.

The next question becomes:

What should we do with low agreement?

7. Disagreement should change control flow

A useful advanced agent does not treat disagreement as a cosmetic statistic.

It changes what happens next.

For example:

def decide_next_step(stats: ConsensusStats) -> str:
    if stats.agreement >= 0.8 and stats.margin >= 0.5:
        return "accept_candidate"

    if stats.agreement >= 0.6:
        return "verify_candidate"

    return "gather_more_evidence"

Now self-consistency becomes part of the agent architecture.

sample trajectories
        ↓
normalize answers
        ↓
measure agreement
        ↓
high agreement ────────────────┐
        ↓                      │
verify if needed               │
                               │
low agreement                  │
        ↓                      │
acquire evidence / branch      │
        ↓                      │
re-evaluate                    │
                               ↓
                         final decision

The critical step is still verification.

8. Consensus is not evidence

Imagine five samples all say:

The database is down.

Four out of five agree.

That tells us something about the model distribution.

It does not tell us that the database is down.

The database itself can answer that question more strongly.

health = db_health_check()

If the health check says:

healthy

then model consensus loses.

Always.

This gives us the governing rule:

Use consensus to decide where to spend verification effort, not as a replacement for verification.

9. Correlated failures are the central danger

Self-consistency sounds powerful because multiple independent attempts appear to give us independent evidence.

But they may not actually be independent.

Suppose every sample comes from:

• the same model,
• the same prompt,
• the same retrieved context,
• the same hidden assumptions,
• the same training distribution,
• the same missing evidence.

Then the samples can fail together.

This is correlated error.

Example:

sample 1 → assumes API is synchronous → wrong
sample 2 → assumes API is synchronous → wrong
sample 3 → assumes API is synchronous → wrong
sample 4 → assumes API is synchronous → wrong
sample 5 → assumes API is synchronous → wrong

Agreement:

100%

Correctness:

0%

That is why agreement must not be interpreted as statistical independence unless you have evidence that the trajectories are genuinely diverse.

10. Measure trajectory diversity, not just answer diversity

Two trajectories can produce the same final answer through very different reasoning.

Or they can produce different wording while sharing the exact same flawed assumption.

So answer diversity is only one diagnostic.

You may also want to record:

• assumptions,
• evidence used,
• tools called,
• retrieved sources,
• intermediate hypotheses,
• selected strategy,
• final answer.

A structured trajectory might look like:

from dataclasses import dataclass, field


@dataclass
class Trajectory:
    strategy: str
    assumptions: list[str] = field(default_factory=list)
    evidence_ids: list[str] = field(default_factory=list)
    tool_calls: list[str] = field(default_factory=list)
    final_answer: str = ""

Now the runtime can ask:

Did five samples agree because they independently converged?

Or because all five made the same assumption?

That is a much more valuable distinction.

11. Strategy-conditioned sampling

One way to reduce correlated reasoning is to deliberately vary the strategy.

Instead of:

solve this
solve this
solve this
solve this

use different reasoning contracts.

For example:

sample 1 → solve directly
sample 2 → look for counterexamples
sample 3 → identify hidden assumptions first
sample 4 → derive from constraints
sample 5 → verify with environment/tools first

In code:

STRATEGIES = [
    "direct",
    "counterexample",
    "assumption_first",
    "constraint_first",
    "evidence_first",
]

Then:

def generate_diverse_trajectories(model, task: str) -> list[Trajectory]:
    trajectories = []

    for strategy in STRATEGIES:
        trajectories.append(
            model.reason(task=task, strategy=strategy)
        )

    return trajectories

This is not guaranteed to produce independent errors.

But it makes diversity an explicit design variable instead of hoping temperature will create it accidentally.

12. Temperature is not a diversity strategy

Increasing sampling temperature may produce more variation.

But variation is not necessarily useful diversity.

You may get:

• different wording,
• different ordering,
• more speculative content,
• more malformed outputs,
• more random mistakes.

Useful diversity should vary something meaningful about the solution path.

For example:

hypothesis
source selection
algorithm
tool sequence
constraint interpretation
failure diagnosis

Measure those differences directly where possible.

13. Self-consistency vs Best-of-N

The two techniques look similar.

Both generate multiple candidates.

But the selection logic differs.

Best-of-N

generate candidates
        ↓
score each candidate
        ↓
choose highest score

Self-consistency

generate reasoning trajectories
        ↓
normalize outcomes
        ↓
measure convergence / disagreement
        ↓
aggregate or escalate

Best-of-N asks:

Which candidate looks best?

Self-consistency asks:

Do independent attempts converge on the same outcome?

They can be combined.

14. Self-consistency + verification

A stronger architecture is:

sample N trajectories
        ↓
cluster outcomes
        ↓
identify leading hypothesis
        ↓
run external verification
        ↓
PASS → accept
FAIL → inspect next hypothesis
UNKNOWN → gather more evidence

This is more useful than blindly returning the plurality answer.

For example:

def resolve_with_verification(
    ranked_answers: list[str],
    verify,
):
    for answer in ranked_answers:
        result = verify(answer)

        if result == "PASS":
            return answer

        if result == "UNKNOWN":
            continue

    return None

Consensus now helps prioritize verification effort.

It does not substitute for reality.

15. Self-consistency + search

Self-consistency can also guide tree search.

Suppose several reasoning samples independently favor the same branch.

That convergence can become one search heuristic.

root
 ├─ A  ← selected by 6 trajectories
 ├─ B  ← selected by 2 trajectories
 └─ C  ← selected by 1 trajectory

You might allocate more expansion budget to A.

But again:

agreement ≠ branch value

Environment evidence should override model convergence when available.

This distinction becomes especially important when we later move into Tree of Thoughts and Monte Carlo Tree Search.

16. When majority vote works well

Simple majority aggregation is strongest when:

1. the output space is discrete,
2. answers are easily normalized,
3. individual errors are not strongly correlated,
4. objective verification is expensive or delayed,
5. sampling cost is acceptable.

Examples can include:

intent classification
routing decisions
small discrete diagnosis sets
multiple-choice reasoning
canonical entity selection
binary policy decisions

Even here, benchmark it.

Do not assume voting helps simply because it sounds statistically sensible.

17. When majority vote is weak

Majority voting is much weaker when:

• the answer space is open-ended,
• there are many equivalent formulations,
• all samples share the same missing evidence,
• the model has a strong systematic bias,
• the task has rare-but-critical alternatives,
• correctness requires interaction with the environment,
• one unusual minority hypothesis may be correct.

Consider incident diagnosis.

Eight samples say:

memory leak

Two say:

connection pool exhaustion

A pure majority vote chooses memory leak.

But logs may immediately show:

active_connections = max_connections

The minority hypothesis wins because it is externally verified.

18. Disagreement is often more useful than consensus

A high disagreement rate is not merely a failure.

It can be an uncertainty signal.

Suppose:

A: 4 votes
B: 4 votes

A weak runtime says:

retry

A stronger runtime asks:

What observation would distinguish A from B?

That turns disagreement into an evidence-acquisition problem.

For example:

def next_evidence_question(a: Hypothesis, b: Hypothesis) -> str:
    ...

Then acquire evidence from:

• tests,
• logs,
• repository history,
• database queries,
• source documents,
• browser state,
• external APIs,
• deterministic computation.

This is where self-consistency becomes genuinely agentic.

19. Application: coding agents

A coding agent may generate several diagnoses for a failing test.

1. stale cache
2. async race
3. fixture isolation problem
4. stale cache
5. fixture isolation problem

Do not simply vote.

Instead use the distribution to choose discriminating experiments.

hypothesis: stale cache
verification: disable cache / inspect invalidation

hypothesis: fixture isolation
verification: run test alone vs suite

hypothesis: async race
verification: repeated seeded run / timing instrumentation

Self-consistency is useful here because it exposes uncertainty before the agent edits code.

That can prevent premature patching.

A practical coding-agent flow becomes:

failing test
    ↓
sample diagnoses
    ↓
cluster hypotheses
    ↓
choose discriminating test
    ↓
collect evidence
    ↓
select diagnosis
    ↓
patch
    ↓
run verification suite

20. Application: code review agents

Suppose several review passes inspect the same diff.

They might produce:

review 1 → concurrency issue
review 2 → no issue
review 3 → missing authorization check
review 4 → concurrency issue
review 5 → missing authorization check

Consensus alone is not enough.

But repeated independent detection can help prioritize investigation.

A useful review system records:

• defect category,
• file/line,
• supporting evidence,
• reproduction condition,
• confidence,
• overlap with other reviewers.

Then the runtime can distinguish:

three reviewers repeating one vague suspicion

from:

three reviewers independently pointing to the same concrete invariant violation

The second is much stronger.

21. Application: research agents

A research agent may generate multiple interpretations of the evidence.

For example:

hypothesis A → paper supports causal effect
hypothesis B → paper supports correlation only
hypothesis C → evidence insufficient

Self-consistency can reveal that the interpretation is unstable.

But the correct response is usually not more voting.

It is source inspection.

disagreement
    ↓
identify disputed claim
    ↓
open primary source
    ↓
extract exact methodology/result
    ↓
update hypotheses

For research agents, disagreement is particularly valuable because it can identify which claims need stronger citation-level verification.

22. Application: data and analytics agents

Suppose an agent is asked:

Why did conversion fall last week?

Different trajectories might suggest:

traffic mix changed
checkout regression
pricing change
tracking bug
seasonality

Voting is almost meaningless here.

The system should convert those candidate explanations into queries.

traffic mix changed
    ↓
compare acquisition-channel distribution

checkout regression
    ↓
compare checkout error rates

tracking bug
    ↓
compare server events with analytics events

Self-consistency helps enumerate plausible explanations.

The warehouse decides which explanation survives.

23. Application: browser and UI agents

Browser agents often face ambiguous page state.

Several reasoning samples may disagree about whether:

• a modal is open,
• checkout completed,
• a save button succeeded,
• authentication expired,
• navigation changed context.

The correct response is not to vote on page state.

Use the DOM, URL, accessibility tree, network response, or visible confirmation element.

Consensus can decide which state hypotheses need checking.

The browser remains the authority.

24. Application: planning and scheduling

Multiple planning trajectories may propose different schedules.

Here the outputs may all be valid.

Self-consistency is therefore not necessarily the right selection mechanism.

You may instead need optimization against explicit constraints:

cost
lateness
resource conflicts
travel time
risk

This is a useful boundary:

When multiple answers can all be valid but have different utility, use scoring or optimization rather than consensus.

Self-consistency is best when convergence itself contains useful information.

25. Application: support and operations

A support agent may classify a customer issue.

Five trajectories might return:

billing
billing
account_access
billing
refund

A high billing consensus may be useful for routing.

But before taking an irreversible action such as refunding money, stronger checks should apply.

For example:

classification consensus
       ↓
retrieve account state
       ↓
check order/payment facts
       ↓
authorization policy
       ↓
execute or escalate

Consensus may help choose the workflow.

It should not bypass transactional controls.

26. Self-consistency as uncertainty estimation

One of the most useful interpretations of self-consistency is not voting at all.

It is empirical uncertainty estimation.

If repeated trajectories converge strongly, the model distribution is relatively stable for that task/context.

If they diverge strongly, the model distribution is unstable.

That gives us a runtime signal:

low disagreement
       ↓
cheap path

high disagreement
       ↓
expensive path

For example:

if agreement >= 0.85:
    route = "cheap_verifier"
elif agreement >= 0.60:
    route = "strong_verifier"
else:
    route = "gather_evidence_then_frontier_model"

This turns self-consistency into adaptive compute allocation.

That connects directly to the larger advanced-agent architecture.

27. Escalation based on disagreement

A production design might look like:

cheap model × 3
      ↓
strong agreement?
   /       \
 yes       no
  ↓         ↓
verify    gather evidence
  ↓         ↓
finish    cheap model × 5
             ↓
       still disagree?
          /      \
        no        yes
        ↓          ↓
      verify   stronger model
                  ↓
                verify

The point is not the exact thresholds.

The point is that uncertainty changes compute allocation.

Advanced agents should spend expensive inference where it has expected value.

28. How many samples should you generate?

There is no universal answer.

The useful N depends on:

• task difficulty,
• model variance,
• sample correlation,
• call cost,
• latency budget,
• verification cost,
• consequence of error.

So sweep it.

For example:

N = 1
N = 3
N = 5
N = 8
N = 12

Measure:

verified success
agreement calibration
latency
model calls
cost
false confidence

Do not assume N=20 is better than N=5.

If the samples are highly correlated, extra calls can provide almost no new information.

29. Marginal information per sample

A useful diagnostic is whether new samples actually change the distribution.

Suppose:

N=3  → A A A
N=5  → A A A A A
N=8  → A A A A A A A A

If external verification shows A is usually correct, additional samples may be unnecessary.

If external verification shows A is sometimes systematically wrong, more identical samples are also unnecessary.

Either way, the marginal value of extra sampling is low.

So log the marginal information gain from additional samples.

You do not need a perfect information-theoretic estimator.

Simple changes in:

• winner,
• agreement,
• unique hypotheses,
• evidence diversity,
• strategy diversity

can already reveal whether additional calls are doing useful work.

30. Consensus calibration

Suppose the runtime sees 90% agreement.

How often is that majority actually correct?

That is an empirical question.

Build a calibration table.

For example:

agreement bucket   verified accuracy
0.50–0.59          61%
0.60–0.69          68%
0.70–0.79          76%
0.80–0.89          84%
0.90–1.00          88%

This immediately reveals something important.

Even 100% model agreement may not imply 100% correctness.

The gap between agreement and verified accuracy measures systematic correlated failure.

31. A simple calibration record

from dataclasses import dataclass


@dataclass
class ConsistencyRun:
    task_id: str
    sample_count: int
    winner: str
    agreement: float
    margin: float
    unique_answers: int
    verified_correct: bool | None
    latency_ms: float
    cost: float

Store these runs.

Then aggregate them later.

The goal is not merely to say:

self-consistency helped

The goal is to know:

on which tasks
at which agreement levels
with which N
at what cost
under which verification regime

32. Correlation diagnostics

If self-consistency performs poorly, inspect correlation.

Useful questions:

• Do samples use the same retrieved documents?
• Do they make the same initial assumption?
• Do they call the same tools in the same order?
• Do they choose the same strategy label?
• Do they fail on the same test cases?
• Do different models make the same error?

A simple overlap metric can already help.

For example:

def jaccard(a: set[str], b: set[str]) -> float:
    if not a and not b:
        return 1.0

    return len(a & b) / len(a | b)

You might use this for:

evidence IDs
tool calls
assumption labels
retrieved document IDs

High overlap plus high agreement is not the same as independent convergence.

33. Multiple models can reduce some correlation

You can diversify across models:

local model
frontier model
code specialist
reasoning specialist

This can reduce some shared failure modes.

But do not assume models are independent simply because they have different names.

They may still share:

• training data,
• common benchmarks,
• prompt structure,
• retrieved evidence,
• system assumptions.

Model diversity is another hypothesis to test.

It is not an independence certificate.

34. Specialist disagreement is often more informative

Imagine three agents:

security reviewer
performance reviewer
correctness reviewer

They are not expected to produce identical answers.

Consensus is therefore the wrong objective.

Instead, you want coverage.

This is the distinction between self-consistency and multi-agent specialization.

self-consistency
    → multiple attempts at broadly the same decision

specialist ensemble
    → intentionally different decision functions

If specialists agree, that may be interesting.

But disagreement may be exactly what the architecture is designed to produce.

35. Consensus collapse

A common failure is what we can call consensus collapse.

The runtime adds more and more model calls, but the samples are functionally identical.

Example:

trajectory 1 → same retrieval → same assumption → same answer
trajectory 2 → same retrieval → same assumption → same answer
trajectory 3 → same retrieval → same assumption → same answer
trajectory 4 → same retrieval → same assumption → same answer

The system appears highly confident.

But it has not explored anything.

Diagnostics:

unique answer ratio
unique strategy ratio
unique evidence ratio
unique tool-path ratio

If all four approach zero, self-consistency has become repeated inference, not meaningful sampling.

36. Do not optimize for disagreement either

The opposite mistake is deliberately forcing every sample to be different.

That can create artificial diversity.

You do not want:

five arbitrary opinions

You want:

multiple plausible solution paths

The objective is not maximal disagreement.

It is enough diversity to reveal uncertainty and enough convergence to exploit repeated evidence.

37. A complete minimal runtime

Here is a compact architecture:

from collections import Counter
from dataclasses import dataclass
from typing import Callable


@dataclass(frozen=True)
class Sample:
    strategy: str
    answer: str
    evidence_ids: tuple[str, ...] = ()


@dataclass(frozen=True)
class ConsistencyDecision:
    answer: str | None
    agreement: float
    margin: float
    samples: tuple[Sample, ...]
    action: str


def choose_consistency_action(
    samples: list[Sample],
    verify: Callable[[str], str],
) -> ConsistencyDecision:
    if not samples:
        raise ValueError("samples must not be empty")

    counts = Counter(sample.answer for sample in samples)
    ranked = counts.most_common()

    winner, winner_votes = ranked[0]
    second_votes = ranked[1][1] if len(ranked) > 1 else 0

    total = len(samples)
    agreement = winner_votes / total
    margin = (winner_votes - second_votes) / total

    verification = verify(winner)

    if verification == "PASS":
        return ConsistencyDecision(
            answer=winner,
            agreement=agreement,
            margin=margin,
            samples=tuple(samples),
            action="accept_verified",
        )

    if verification == "FAIL":
        return ConsistencyDecision(
            answer=None,
            agreement=agreement,
            margin=margin,
            samples=tuple(samples),
            action="inspect_alternative_hypotheses",
        )

    if agreement < 0.60:
        action = "gather_more_evidence"
    else:
        action = "escalate_verification"

    return ConsistencyDecision(
        answer=None,
        agreement=agreement,
        margin=margin,
        samples=tuple(samples),
        action=action,
    )

Notice what this runtime does not do.

It does not automatically trust the winner.

Consensus influences routing.

Verification controls acceptance.

38. What should be logged?

At minimum:

task ID
model / model version
prompt version
sampling parameters
strategy per sample
raw answer
normalized answer
supporting evidence IDs
agreement
margin
unique-answer count
verification result
final action
latency
cost

For systems with tools, also log:

tool path
tool outputs / state IDs
failure states

This is how you later distinguish:

model variance
from
retrieval variance
from
verification variance
from
runtime policy errors

39. Debugging: the agent gives different answers every time

Start with the distribution.

Do not immediately turn temperature to zero.

Ask:

Are the answers semantically different?

Are they different only in wording?

Do they use different evidence?

Does one answer consistently verify better?

Is the task itself under-specified?

Then decide whether you have:

• harmless presentation variance,
• useful hypothesis diversity,
• unstable reasoning,
• missing evidence,
• ambiguous requirements.

Different causes need different fixes.

40. Debugging: all samples agree but are wrong

This is the classic correlated-failure case.

Inspect:

1. shared assumptions,
2. shared retrieval,
3. shared prompt framing,
4. shared model family,
5. missing external evidence.

Then add different information, not merely more samples.

For example:

run a test
open the source
query the database
inspect logs
use a different retrieval query
introduce a counterexample strategy

More identical inference is unlikely to repair a systematic error.

41. Debugging: agreement is always low

Low agreement can mean:

• genuinely hard task,
• under-specified task,
• weak model,
• poor answer normalization,
• excessively stochastic sampling,
• too many plausible valid outcomes.

Before adding more samples, determine which one applies.

If multiple answers are all valid, self-consistency may simply be the wrong mechanism.

Use scoring, utility optimization or external constraints instead.

42. Debugging: self-consistency is too expensive

The first fix is not necessarily fewer samples.

Try an adaptive sample budget.

Start with three.

If they strongly agree, stop.

If they disagree, sample more.

For example:

samples = generate(3)
stats = consensus([s.answer for s in samples])

if stats.agreement < 0.67:
    samples.extend(generate(2))

You can continue only while new samples materially change the decision.

This makes sampling conditional rather than fixed.

43. Debugging: majority voting makes results worse

Measure two quantities separately:

oracle@N

Did at least one sample contain the correct answer?

and:

consensus selection accuracy

Did the aggregation rule choose it?

If oracle@N increases but final accuracy falls, the generator is producing useful alternatives and the aggregator is selecting badly.

That tells you exactly where to improve the system.

This is the same evidence-first discipline we used in Best-of-N.

44. A useful experiment

Compare:

A. one-shot reasoning
B. 3-sample majority
C. 5-sample majority
D. 5 strategy-conditioned samples
E. 5 samples + external verification
F. adaptive sampling + external verification

Measure:

verified task success
false-consensus rate
agreement calibration
oracle@N
model calls
latency
cost
cost per verified success

Do not report only answer agreement.

That measures consistency, not utility.

45. False consensus rate

A particularly useful metric is:

false consensus rate

Define it as the fraction of high-agreement cases that fail external verification.

For example:

false_consensus = (
    high_agreement_and_wrong
    / high_agreement_total
)

This tells you how dangerous it is to treat consensus as confidence.

If the number is high, your samples are strongly correlated or your normalization is hiding important differences.

46. Useful metrics

A production self-consistency system should consider tracking:

Outcome metrics

verified task success
false-pass rate
false-consensus rate
cost per verified success

Sampling metrics

sample count
unique-answer ratio
unique-strategy ratio
unique-evidence ratio

Consensus metrics

winner agreement
top-two margin
entropy / distribution spread

Calibration metrics

accuracy by agreement bucket
accuracy by margin bucket

Runtime metrics

model calls
tool calls
latency
tokens
cost

The mechanism should earn its operational cost.

47. The deeper pattern

The important lesson is not:

Ask the model five times.

It is:

Treat repeated reasoning as a noisy measurement process whose distribution can inform runtime control.

That changes the architecture.

single answer
    ↓
probabilistic claim

becomes:

sample distribution
      ↓
uncertainty signal
      ↓
verification / evidence acquisition / escalation
      ↓
validated decision

That is a much more powerful idea.

48. When should you use self-consistency?

Use it when:

• single-run variance is materially affecting quality,
• multiple reasoning paths can plausibly converge,
• outputs can be normalized meaningfully,
• disagreement is useful for routing or escalation,
• extra inference is cheaper than the cost of a wrong decision,
• external verification can still arbitrate important outcomes.

Do not use it merely because you are building an “advanced agent.”

49. When should you not use it?

Avoid or simplify self-consistency when:

• a deterministic tool can answer the question directly,
• one model call already verifies reliably,
• the output space has many equally valid answers,
• all samples share one obvious missing piece of evidence,
• latency matters more than marginal reliability,
• the action is irreversible and still lacks external verification,
• sampling produces cosmetic rather than strategic diversity.

In those cases, another mechanism is usually better.

50. The escalation rule

Use the smallest mechanism that matches the failure.

single answer unstable?
        ↓
measure variance
        ↓
multiple plausible reasoning trajectories?
        ↓
self-consistency
        ↓
low agreement?
        ↓
acquire evidence
        ↓
still ambiguous?
        ↓
search / stronger verifier / specialist routing

Do not jump straight from variance to a ten-agent committee.

51. The application map

The pattern is the same:

consensus helps prioritize
external state decides

52. What comes next

Self-consistency still samples complete reasoning trajectories independently.

The next escalation is more powerful.

Instead of waiting until each trajectory is complete, we can branch during reasoning.

At an intermediate state:

current thought/state
      ↓
  ┌───┼───┐
  A   B   C
  ↓   ↓   ↓
evaluate partial paths
  ↓   ↓   ↓
expand promising branches

That is the bridge to Tree of Thoughts.

It changes the question from:

Which completed reasoning trace should I trust?

into:

Which partial reasoning states deserve more compute?

That is where the next post goes.

Final rule

Self-consistency is useful because disagreement exposes uncertainty and convergence can help allocate compute.

But never forget the evidence boundary:

Consensus tells you what the model distribution believes. Verification tells you what the world supports.

Advanced agents need both.