Part V — Debugging AI-Assisted Development and Research

The diagram is beautiful — and over budget by 250 milliseconds

Chapter 26 verified the agent saw the files. Now it proposes a profile endpoint fanning out to three services per request, with a sequence diagram reviewers call “clean.” The p99 budget is 200 ms. The line items sum to 450. Eloquence is not a load test.

OBSERVATION: the design document claims “meets latency goals” while listing per-hop latencies (auth 80 ms, profile 190 ms, recommendations 180 ms) that sum past the stated 200 ms budget before queueing. HYPOTHESIS H1 (constraint violation): the design contradicts its own numbers — no implementation of this shape can meet budget. H2 (constraint underspecification): the budget was never pinned, so “meets goals” is unverifiable prose rather than a checkable claim. H3 (tradeoff blindness): a viable alternative (cached reads, collapsed hops) exists but was never generated or compared. INFERENCE: none yet — H1/H2/H3 predict different rows in a constraint table and are separable only by one.

This chapter’s question: does the proposed design satisfy its load, latency, and consistency constraints — as arithmetic, not as prose?

Why “ask for a simpler design” fails first

The obvious move — requesting a simpler or cheaper variant — fails because it re-samples eloquence without an oracle. Four defects hide behind design re-rolls:

1. Unpriced opinions. Without a constraint table, “simpler” means fewer boxes in the diagram. Box count is not latency, cost, or blast radius.
2. Missing rejected alternatives. The design presents one shape with no compared options. A single proposal cannot be a tradeoff — tradeoffs require ≥2 priced candidates.
3. Consistency hand-waving. “Eventually consistent” asserted without a staleness bound, conflict rule, or read-repair path. The failure mode ships as a footnote.
4. Load-test postponement. “We’ll benchmark in staging” defers arithmetic that fits on an envelope. Budgets exist to kill designs before they are built.

OPINION: a design without line-item budgets is a mood board. Debugging starts by converting adjectives (“fast,” “scalable,” “resilient”) into rows with numbers and verdicts.

The mental model: designs are constraint-satisfaction claims — debug them like proofs, line by line. Every architectural sentence of the form “X handles Y” owes a quantity, a bound, and a failure behavior when the bound is exceeded.

There is a specific reason not to trust the design’s own arithmetic. Dziri and colleagues showed that transformers solve multi-step compositional problems — multi-digit multiplication, logic puzzles, dynamic programming — by pattern-matching against similar cases rather than by computing, and that accuracy collapses as the problem’s dependency graph grows wider or deeper (Dziri et al., 2023). Adding per-hop latencies with overlap and queueing, or multiplying fan-out by request rate against a quota, is exactly that kind of task. The prose “this meets the budget” is a compositional claim reasoned in prose by a system that is unreliable at exactly that. Be precise about the bound: a scratchpad or a bigger model helps in-distribution but does not fix the depth problem, whereas a model that executes the arithmetic — a code interpreter, a spreadsheet formula, a queueing calculator — is reliable at it, and even frontier models fail large-number arithmetic without one. So the fix is not merely “add it up yourself” — it is to make the design present its arithmetic as executed computation rather than asserted prose. That is exactly what the constraint table below forces.

The method: the constraint table and the tradeoff matrix

Freeze the design, then price it. The constraint table lists each binding requirement as a checkable row:

    flowchart TD
    FR["freeze the design; quote its own per-hop numbers"] --> CT["constraint table: bound + line-item arithmetic + verdict per row"]
    CT --> Q{"any row's verdict?"}
    Q -->|"FAIL under the design's own numbers"| RS["reject the design SHAPE regardless of prose (H1)"]
    Q -->|"UNVERIFIABLE — bound or failure behavior unstated"| RD["reject the design DOCUMENT regardless of shape (H2)"]
    Q -->|"all PASS"| TM["tradeoff matrix: >=2 candidates priced on identical rows"]
    TM --> C{">=2 priced alternatives present?"}
    C -->|no| H3["H3: single-shape presenting — generate and price alternatives"]
    C -->|yes| AC["advance the passing candidate to prototype, with the table as the load-test oracle"]
  

CONSTRAINT TABLE (design frozen; arithmetic only):
constraint | bound | design line-items | verdict
p99 latency | < 200ms | 80 + 190 + 180 = 450ms sequential (no overlap stated) | FAIL (H1)
peak load | 100 rps | 3 fan-out reads x 100 = 300 downstream rps vs quota 250 | FAIL (H1)
consistency | read-your-write, 5s bound | async invalidation, no bound stated | UNVERIFIABLE (H2)
blast radius | 1 AZ loss tolerable | all reads single-AZ pinned | FAIL (H1)
RULE: one FAIL row rejects the design shape regardless of prose quality.
One UNVERIFIABLE row rejects the design document regardless of shape.

Then demand the tradeoff matrix: ≥2 alternatives priced on the same rows with rejected options named and priced (not dismissed as “less elegant”):

OBSERVATION (constructed illustration, not a measured run): pricing the frozen proposal convicted H1 on two rows (latency, load) and H2 on one (consistency unbounded); candidate B passed load and latency only by explicit staleness tradeoff the original document never stated. UPDATED BELIEF: H1 supported (design-shape failure) with H2 live (document failure) for this instance; H3 exonerated once candidate B was priced — the alternative existed, it was the comparison that was missing.

Example: pricing the fan-out before staging

The design paragraph claimed “parallel fan-out keeps latency flat.” The table tests the sentence:

# design arithmetic: prose frozen, numbers checked (no staging needed for rejection)
budget_p99 = 200  # ms, from intent contract (Ch25); UNKNOWN if contract lacks it
hops = {"auth": 80, "profile": 190, "reco": 180}  # design's own stated per-hop p99
sequential = sum(hops.values())  # 450ms — OBSERVATION: design's numbers, added up
# HYPOTHESIS H1: shape violates budget. Prediction: even fully parallel execution
# needs max(hops)=190ms + overhead; design states no overlap, no cache, no deadline.
# Discriminator: require the design to state overlap/deadline/fallback per hop, then
# re-price. If re-priced total still exceeds budget -> H1 convicted for this shape.
parallel_floor = max(hops.values())  # 190ms before overhead, queueing, retries
print("sequential:", sequential, "parallel floor:", parallel_floor, "budget:", budget_p99)
# Consistency probe: demand staleness bound + conflict rule + read path during
# invalidation lag. Absent -> H2 (unverifiable), document rejected regardless of shape.

Rejection here costs an envelope, not a staging environment. Any single FAIL row vetoes the shape; any UNVERIFIABLE row vetoes the document. Only a fully PRICED-and-PASSING table advances to prototyping — and even then as one candidate among priced alternatives, never as the anointed shape.

No vendor claim (“handles 10× scale”), confidence statement, or single approving review substitutes for priced rows. Downstream-symptom relief (“staging p99 looked okay once”) is not a budget proof.

Research lineage: this is a lightweight ATAM

The constraint table and tradeoff matrix are decades old. The Software Engineering Institute’s Architecture Tradeoff Analysis Method (ATAM) evaluates a proposed architecture by turning stakeholder quality goals into concrete scenarios with measurable response bounds, organizing them into a utility tree, and then locating the design’s sensitivity points (where a decision strongly affects a quality attribute), tradeoff points (where it affects several, in tension), and risks (Kazman, Klein & Clements, 2000). This chapter’s constraint table is a utility tree with the scenarios already quantified; the tradeoff matrix is ATAM’s tradeoff-point analysis compressed to a spreadsheet. Recent work even uses LLMs to help run ATAM — which only sharpens the point that the model assists the analysis, it does not replace the arithmetic.

“Rejected alternatives named” is the ADR discipline. An Architecture Decision Record captures the context, the decision, the alternatives considered, and the consequences, precisely so that a future reader can see the design was a choice among priced options rather than the first plausible shape (Nygard, 2011). A single-candidate AI design is an ADR with the “alternatives” section deleted.

Lab 27: price-and-compare with pre-written verdicts (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own priced tables.

Setup. Take one AI-generated design (API, schema, or architecture note) with its constraint sources (intent contract, quota docs, SLO statements) pinned. Freeze the design text and hashes. The pricing is the independent variable; design text, bounds, and per-hop numbers are controlled.

Task.

1. Before pricing, write H1/H2/H3 with distinct predicted table patterns: H1: “≥1 FAIL row under the design’s own numbers”; H2: “≥1 UNVERIFIABLE row (bound or behavior unstated)”; H3: “exactly one candidate proposed, no priced alternative.”
2. Build the constraint table (load, latency, consistency, blast radius minimum) with line-item arithmetic quoted from the design, then the tradeoff matrix with ≥2 candidates on identical rows.
3. Record OBSERVATION (row verdicts verbatim) and UPDATED BELIEF per hypothesis. A redesign proposed without re-pricing is explicitly not evidence; re-price or UNKNOWN.

Success criterion. A completed constraint table plus a ≥2-candidate tradeoff matrix with per-row verdicts and an explicit accept/reject per candidate. A reworded design without tables is explicitly not completion.

Companion tool: Design Claim Verifier

What it accepts: the frozen design text, the bound sources (contract predicates, quota/SLO references with hashes), per-hop numbers quoted from the design, and the candidate set. What it performs: it blocks design acceptance while any load/latency/consistency row is UNVERIFIABLE, checks line-item arithmetic against bounds, requires ≥2 priced candidates with rejected options named, and stamps the verdict per revision hash. What it can establish: whether the proposed shape satisfies its stated constraints under its own numbers — and which row rejects it, for the examined revision only. What it cannot establish: implementation fidelity (a passing design can still be built wrong), real-world load behavior beyond the stated numbers, or bound correctness (a foolish budget precisely met is precisely foolish). It never treats eloquence, confidence, vendor claims, or single staging observations as diagnosis. How its output changes your next action: FAIL routes to re-shaping with re-pricing; UNVERIFIABLE routes to bound/behavior specification; single-candidate routes to alternative generation; all-passing routes to prototype under the table as the test oracle.

Paper form, sufficient for this chapter:

Design hash: ___  Bounds source: ___ (hash ___)
Latency: bound ___ | items ___ = ___ | PASS/FAIL/UNVERIFIABLE
Load: bound ___ | items ___ = ___ | PASS/FAIL/UNVERIFIABLE
Consistency: bound ___ | staleness ___ | conflicts ___ | PASS/FAIL/UNVERIFIABLE
Blast radius: bound ___ | shape ___ | PASS/FAIL/UNVERIFIABLE
Candidates priced: ___  ACCEPT: ___  REJECT: ___ (row: ___)

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Price before prototyping.

Reusable procedure: arithmetic before architecture opinions

1. Freeze the design — text, revision hash, bound sources.
2. Tabulate constraints — load, latency, consistency, blast radius with bounds.
3. Price line items — the design’s own numbers, added up, verdict per row.
4. Compare candidates — ≥2 shapes on identical rows, rejections named by row.
5. Pin the table — surviving bounds become build-time assertions and load-test oracles.

Failure modes

• Eloquence-as-evidence. Fluent justification filed as satisfaction. Adjectives are not arithmetic.
• Single-shape presenting. One proposal with no alternatives. Comparison requires competitors.
• Boundless consistency. “Eventual” without seconds, conflicts, or repair paths. Unbounded claims are UNVERIFIABLE by construction.
• Staging instead of arithmetic. Deferring envelope math to load tests. Staging measures the built shape; tables reject the unbuildable one for free.
• Vendor-claim import. “The managed cache handles 1M rps” pasted as a priced row. Vendor claims are leads with citations, not line items (mark changeable facts; verify before building).
• Single-run load theater. One green staging run certifying the design. Load behavior needs distributions across trials, not anecdotes.

Limits, per contract: one verification covers one design revision under one bound set; it does not certify implementations, does not validate the bounds’ wisdom, and does not survive bound changes without re-pricing. UNKNOWN where per-hop numbers come from prose rather than quoted quantities.

References

• Rick Kazman, Mark Klein, and Paul Clements. ATAM: Method for Architecture Evaluation. Technical Report CMU/SEI-2000-TR-004, Software Engineering Institute, 2000. https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=5177
• Michael Nygard. Documenting Architecture Decisions. 2011. https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions
• Nouha Dziri, Ximing Lu, Melanie Sclar, et al. Faith and Fate: Limits of Transformers on Compositionality. Advances in Neural Information Processing Systems 36 (NeurIPS), 2023. https://arxiv.org/abs/2305.18654

Debugging Checklist

• Design frozen (text + revision hash + bound sources)?
• Constraint table complete (load/latency/consistency/blast-radius with bounds)?
• Line-item arithmetic quoted from the design (not from memory)?
• Every row verdict PASS/FAIL/UNVERIFIABLE (no blanks, no “roughly”)?
• ≥2 candidates priced on identical rows with rejections named by row?
• H1/H2/H3 FORECASTs pre-written with distinct predicted patterns?
• No eloquence, confidence, vendor claim, or single staging run cited as proof?

What This Chapter Established

• Constraint-first design debugging: the constraint table (arithmetic veto per row) plus the tradeoff matrix (≥2 priced candidates), demonstrated on the constructed 450-ms fan-out case.
• The price-before-prototype intervention with pre-written H1/H2/H3 verdict patterns; no measured runs claimed.
• Lab 27 as a proposed pricing record the reader executes; the Design Claim Verifier contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any real system’s performance, any bound’s correctness, or any implementation’s fidelity. One revision priced; nothing built or certified.
• Research grounding: the constraint table + tradeoff matrix is a lightweight ATAM (quality-attribute scenarios, sensitivity/tradeoff points, risks — Kazman et al.), and “rejected alternatives named” is the ADR discipline (Nygard); the design’s own “meets budget” prose is a compositional claim reasoned in prose, and transformers are measurably unreliable at that (Dziri et al.) while being reliable when the arithmetic is executed — so demand the computation, shown, not asserted.
• Forward link: priced designs still ship with literature sections claiming prior art the author never opened. Prose about systems was checked; prose about sources is Chapter 28’s jurisdiction.

Next

The architecture is priced and passing — and its “related work” section cites three papers, one of which says the opposite of what is claimed and another of which cannot be found at all. Constraint arithmetic cannot catch invented provenance. Chapter 28, “Debugging AI Research,” resolves every reference to a retrieved byte: citation integrity as evidence hygiene, with the fabricated-reference drill.