More Is Not Better

The seductive assumption runs like this. Useful information is good. The model has spare capacity. Therefore adding more potentially useful information should be better, or at worst harmless. Teams act on it constantly: the whole repository goes into the window just in case, every retrieved document is kept because retrieval must have returned it for a reason, history is never truncated because something old might matter. The budget chapter showed this costs money and headroom. This chapter shows it costs something more fundamental. Past a point, added context does not merely cost. It injures.

The injury takes several distinct forms, and the chapter’s first obligation is to keep them distinct. A request can fail because it overflowed hard capacity. It can fail because it is bloated with unnecessary material. It can fail because added material interferes with the use of relevant material. It can fail because relevant material sits in a position the model uses poorly. These are different mechanisms with different remedies, and the literature of the last three years has measured each of them. This is the book’s first chapter where empirical work visibly constrains the argument, and the constraint is welcome: it converts suspicion about long contexts into an inventory of named, measured failure modes.

Four claims, not one

Before the evidence, the ladder it climbs. “The model handles the context” conflates four claims that must be separated:

information is present
        ↓
the model can locate it
        ↓
the model can use it
        ↓
using it improves the final task

Presence is a property of the bundle, verifiable by inspection. Locatability is retrieval under load: can the model find the needle when the haystack grows? Usability is reasoning under load: found the fact, can it draw the inference? Task improvement is the end-to-end question: did the bundle containing the fact outperform the bundle without it? Each step can fail while the earlier ones succeed. A model can contain the answer, fail to locate it among distractors, locate it but misread it, or use it correctly while the rest of the bundle drags the final answer off course. Evaluations that test only presence, the classic needle tests, pass models that fail every later step. The literature since has climbed the ladder one rung at a time.

The ladder is also a diagnostic procedure. Given a failure, attribute it downward: check presence first by inspecting the bundle, since a surprising number of “reasoning failures” turn out to be absent evidence. If present, test locatability by asking for the fact directly; success here with failure on the full task isolates usability. If usable in isolation, compare the full bundle against the bundle without the fact; improvement missing here isolates task-level interference from the surrounding material. Each test is cheaper than the theory it replaces, and the book’s experiments are built to run exactly this descent rather than stopping at final accuracy.

What Lost in the Middle actually showed

The study the field reaches for first is Liu et al.’s “Lost in the Middle,” and it deserves careful handling because its fame exceeds its scope. The authors tested language models of its era on two tasks requiring identification of relevant information in long inputs: multi-document question answering and key-value retrieval. The observed pattern: performance was highest when relevant information sat at the beginning or end of the context and degraded significantly when models had to access information in the middle, even for models advertised as long-context. The paper was accepted to TACL in 2023.

Stated with the qualifications the book requires: the task population was QA and retrieval, not agentic tool use or multi-step reasoning. The model population was the 2023 generation. The positional pattern is a U-shaped curve, primacy plus recency with a middle trough, not a universal law of all architectures at all lengths. What the study established, and what survives those qualifications, is the mechanism: position within the bundle affects usability independently of presence. The information was in every condition. Behaviour differed anyway. That single dissociation, presence without reliable use, is the founding observation of this chapter, and everything since has refined rather than refuted it.

Beyond the needle: RULER

If Lost in the Middle showed that position matters for retrieval-shaped tasks, RULER asked how far the damage extends when tasks get harder. Hsieh et al.’s benchmark, presented at COLM 2024, expands the vanilla needle test along two axes: varied needle types and quantities, and new task categories, multi-hop tracing and aggregation, that require behaviour beyond searching. Thirteen tasks over seventeen long-context models, with sequence length and complexity configurable.

Two findings matter for this chapter. First, near-perfect vanilla needle scores coexisted with large performance drops as length and task complexity increased. Passing the simple test told the evaluators almost nothing about the harder ones. Second, and more quotable: although every tested model claimed a context size of 32K tokens or more, only half maintained satisfactory performance at 32K. The headline number described capacity. The benchmark measured usability. They diverged at half the advertised length for half the field. “Satisfactory” is the authors’ criterion rather than a universal threshold, and the population is the 2024 generation, but the shape of the finding, claimed length versus demonstrated length, recurs in every study this chapter reviews.

Application tasks diverge: HELMET

HELMET, by Yen et al., presented at ICLR 2025, attacks the problem from the other direction: not harder synthetic tasks but application-centric ones. Seven categories, synthetic recall, long-document QA, summarisation, many-shot in-context learning, retrieval-augmented generation, passage re-ranking, and generation with citations, with controllable lengths to 128K tokens and model-based evaluation for the categories where n-gram overlap misleads. Fifty-nine long-context models in the published study.

Three findings carry weight here. First, synthetic tasks like needle retrieval do not reliably predict downstream performance; rank correlations between synthetic and application categories were poor. A model that aces the needle test has demonstrated a narrow skill, and buyers of long context should stop accepting it as a general certificate. Second, the categories exhibit distinct trends with low mutual correlation: long-context ability is not one thing but several, degrading at different rates. Third, degradation steepens with task complexity, and on the hardest categories, tasks requiring full-context reasoning or complex instruction following, open-source models lagged closed ones with the gap widening as length increased. The study’s population is the 2024–2025 generation and its application tasks remain benchmark proxies rather than live agent traces, but its structural message is the one this book needs: fitting the application into the window is the beginning of the evaluation, not the end.

When the needle stops looking like the question: NoLiMa

Both studies above retain a feature that flatters models: literal overlap between the question and the sought information. Real queries rarely share vocabulary with their answers, and Modarressi et al.’s NoLiMa, presented at ICML 2025, removes the crutch. Its needle set minimises lexical overlap between questions and needles, forcing inference over latent associations: knowing that the Kiasma museum stands in Helsinki, for instance, rather than matching the word “Helsinki.” Thirteen models claiming contexts of at least 128K tokens.

The results are the starkest in this chapter. While models performed strongly at short lengths under 1K tokens, performance degraded significantly as length grew: at 32K, eleven of thirteen models fell below half of their own short-context baselines. GPT-4o, among the strongest, fell from 99.3 per cent to 69.7 per cent. Neither reasoning-capable models nor chain-of-thought prompting rescued performance. The authors’ analysis attributes the decline to attention spreading thin over longer spans when literal matches are absent, which is their proposed mechanism rather than established fact, but the behavioural finding stands independently: remove lexical shortcuts and demonstrated usable length collapses far below claimed length.

NoLiMa also contributes a definitional caution this chapter adopts. Its “effective length,” the longest context at which a model holds at least 85 per cent of its short-context base score, is one reasonable operationalisation among several. RULER’s “satisfactory performance” is another. The book will use the term effective length only with the defining criterion attached, because the literature has no single metric and pretending otherwise would manufacture a consensus that does not exist.

Isolating length itself: the context-rot report

A persistent objection shadows every study so far: longer inputs usually mean harder tasks, so perhaps length is innocent and difficulty guilty. Chroma’s “Context Rot” technical report, by Hong, Troynikov, and Huber in July 2025, is designed to defeat that objection directly. Eighteen models across the GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 families are tested on tasks whose complexity is held fixed while input length alone varies. It is a company technical report rather than peer-reviewed work, with an open codebase for replication, and its model-specific behavioural claims should be read as findings awaiting independent replication. Its structural findings, however, align with and extend the peer-reviewed literature above.

The report’s most useful contribution to this book is a clean distinction the chapter adopts as taxonomy:

• Distractors are topically related to the sought information but do not answer the question.
• Irrelevant content is unrelated to both.

The difference matters behaviourally. Even a single distractor reduced performance relative to a needle-only baseline, and four distractors compounded the damage, with impact varying non-uniformly across distractors and models. Irrelevant content degraded performance too, but through a different channel: sheer volume spreading attention thin. A team that treats all added tokens as equivalent misses both mechanisms. Distractors mislead; volume dilutes. The remedies differ, and the later chapters take them in that order: selection against distractors before budgeting against volume.

Three further findings deserve recording. First, the focused-versus-full comparison: on a cleaned LongMemEval conversational set, prompts containing only relevant material averaged around 300 tokens while full-history prompts averaged around 113K, and every model family performed significantly worse on the full versions. Thinking modes narrowed the gap without closing it. Presence was equalised in the sense that the answers existed in both conditions; locatability under load was not. Second, behavioural styles diverged under ambiguity: Claude models tended to abstain when uncertain while GPT models tended to hallucinate confidently. The styles have different failure signatures, and a team measuring only final accuracy will misread both. Third, and strangest, models consistently performed better on shuffled haystacks than on coherent ones: logical structure in the irrelevant material hurt rather than helped. The authors offer no settled mechanism, and this book adopts the finding with the same reserve: structure sensitivity is observed, unexplained, and a warning against assuming that tidy, well-organised filler is neutral.

Reading the literature as a harness builder

The studies above use synthetic haystacks and question answering, not coding agents with tools. Transferring their findings to the book’s domain requires care, and this section states the transfer as hypotheses rather than conclusions.

Book hypothesis. In coding-agent traces, distractor-shaped material, superseded documentation, near-miss file excerpts, stale tool results, will degrade evidence use more per token than unrelated material of equal length.

The rationale follows the Chroma distinction directly: a superseded configuration document does not merely consume tokens but actively competes for the retrieval that the current document needs, while an unrelated chat transcript mostly dilutes. If the future corpus confirms the gradient, selection machinery earns priority over pure volume budgeting; if it does not, the distinction stays in the synthetic literature and out of the capstone.

Book hypothesis. Position sensitivity will persist in agent-shaped bundles but with a different signature: recency-dominated rather than U-shaped, because tool observations adjacent to the current request compete most directly with it.

Lost in the Middle’s U-curve came from static documents read in order. An agent bundle is rewritten every turn, with fresh observations landing late and standing instructions early. The prediction is that the trough moves: middle-position weakness for static layouts, late-position crowding for agentic ones. Family B of the proposed experiment is designed to detect exactly this difference, which is why it uses several interior positions rather than three.

A third transfer concerns behavioural style. Chroma’s abstain-versus-hallucinate split, Claude caution against GPT confidence, matters for how Context Lab scores outcomes. A harness whose model abstains under ambiguity fails loudly and measurably; one whose model hallucinates fails silently and contaminates session state for later turns. The book’s measurement of unsupported claims exists to catch the second style. Neither style is virtuous in itself; they are different failure signatures requiring different detection, and a single accuracy number conflates them.

The taxonomy this chapter owns

The evidence now supports precise language. Context rot is the book’s engineering label for the observed phenomenon: task performance degrading as input tokens grow, holding task complexity constant. Anthropic’s own documentation now uses the term the same way, noting that accuracy and recall degrade as token count grows and that more context is not automatically better. But rot is the phenomenon, not the explanation. Beneath it, this chapter owns four terms:

• Overflow. The request exceeds hard capacity. Rejected, truncated, or halted. Chapter 4’s territory; mentioned here only as the boundary where soft degradation becomes hard failure.
• Bloat. The bundle contains unnecessary material. Measurable by the Chapter 2 instrument as repetition, stable-but-unexamined standing costs, and admitted-but-unused content. Bloat is the precondition the other three exploit.
• Interference. Added material disrupts use of relevant material. Distractors mislead; volume dilutes; similar-but-wrong candidates capture the retrieval that the right candidate needed. The Chroma distractor experiments are its cleanest demonstration.
• Position sensitivity. Present information is used differently depending on where it sits. The U-shaped middle trough is the classic signature; boundary preference its milder form.

Three neighbouring terms are explicitly deferred, and the deferral is part of the taxonomy’s honesty. Conflict, where two context items push behaviour in incompatible directions, belongs to Chapter 19’s authority machinery; this chapter’s distractors mislead without contradicting. Staleness, where once-valid information decays, belongs to Chapter 20; everything here is evaluated at a single point in time. Loss, where important information was removed or transformed away, belongs to Chapter 11’s compaction analysis; this chapter studies only addition, never removal. A reader who can sort a failure into exactly one of these seven bins has understood the chapter.

The evaluation landscape behind the taxonomy compresses into one comparison. It is organised by what each study isolates, which is the only comparison that advances the argument:

Proposed experiment: evidence use and position

The book’s own contribution must replicate and extend this literature in the coding-agent domain, where tool results, file excerpts, and histories replace synthetic haystacks. The design below is frozen as a proposal; Context Lab does not yet exist, and no result is claimed. It comprises two task families that must never be mixed in a single manipulation.

Family A: critical fact and evidence use. The task requires one specific piece of information to succeed, for example a single configuration value buried in project material without which a repair task cannot pass its checks. Conditions:

A  critical evidence only

B  + irrelevant material (unrelated files, benign history)

C  + plausible but irrelevant material

D  + semantically similar distractors (near-miss values, superseded docs)

The B-to-D gradient mirrors the irrelevant-to-distractor distinction: volume dilution versus active misleading. Distractor corpora are frozen per task so conditions are comparable.

Family B: position. Hold total content as constant as possible while moving the critical evidence through several relative positions, early, two interior points, and late, rather than only three. Content constancy is approximate in practice; report the token deltas per position so readers can judge whether length confounded the manipulation.

Controls, pre-registered: model and exact version; decoding settings including temperature; the full instruction bundle frozen byte-identical; tool availability fixed; the task and its critical evidence; the distractor corpus with versions; the token budget and, for Family B, the evidence position per trial; the number of repeated trials per cell; the success rubric. Randomise or counterbalance trial order and distractor placement where the harness permits.

Measurements, separated: final task success against the external rubric; critical-fact recovery, scored independently of success by checking whether the model’s trace demonstrably used the evidence; unsupported claims, counting assertions in the output with no bundle basis; latency; rendered input size. The recovery-versus-success split is the design’s most important feature: it distinguishes the model that never found the evidence from the model that found it and reasoned incorrectly. Later chapters assign those two failures to different mechanisms, retrieval-side versus reasoning-side, and this measurement is what lets them.

Reading the outcomes, pre-registered: Family A tests the book’s interference hypothesis, that D degrades more than C degrades more than B relative to A at matched token counts. Family B tests position sensitivity in agent-shaped material, including the recency-crowding signature predicted above. If Family A shows no gradient, the honest report is that synthetic-haystack interference did not transfer to this domain, and the selection chapters lose a motivation they must then earn elsewhere. The two families must not be combined into a single manipulation, for example distractors placed only in middle positions, unless the design is explicitly factorial with position crossed against distractor type; otherwise a position effect will masquerade as interference or the reverse, and the taxonomy this chapter built will be blurred by its own experiment. A proposed experiment that cannot surprise its author is decoration.

Where this leaves the problem

Fit is necessary and nowhere near sufficient. The bundle that fits can still be bloated, interfered with, and badly arranged, and the literature shows each failure arriving well below the advertised limit: half the RULER field faltering at 32K, eleven of thirteen NoLiMa models halved at the same length, focused-to-full gaps in every family Chroma tested. The seductive assumption fails at every rung of the ladder. Presence does not imply locatability, locatability does not imply usability, and usability does not imply task improvement.

One failure mode now stands out as both established and testable. If position changes usability while content stays fixed, then position is a candidate control variable: the hypothesis Chapter 6 will test is that deliberate placement outperforms wherever-it-landed appending. That is the next chapter’s subject:

Even if we know which information belongs in the bundle, where should we put it?

References

• Kamradt, G. “Needle In A Haystack.” Third-party evaluation repository, original runs November 2023 (GPT-4-128K, Claude 2.1); v2 current. Pressure-test sweep over context length by needle depth. https://github.com/gkamradt/needle-in-a-haystack
• Liu, N. F., Lin, K., Hewitt, J., et al. “Lost in the Middle: How Language Models Use Long Contexts.” TACL 2023; arXiv:2307.03172. Peer-reviewed. U-shaped position sensitivity in QA and key-value retrieval. https://arxiv.org/abs/2307.03172
• Hsieh, C.-P., Sun, S., Kriman, S., et al. “RULER: What’s the Real Context Size of Your Long-Context Language Models?” COLM 2024; arXiv:2404.06654. Peer-reviewed. 13 tasks; half the field unsatisfactory at 32K. https://arxiv.org/abs/2404.06654
• Yen, H., Gao, T., Hou, M., et al. “HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly.” ICLR 2025; arXiv:2410.02694. Peer-reviewed. 7 categories to 128K; synthetic scores do not predict downstream. https://arxiv.org/abs/2410.02694
• Modarressi, A., Deilamsalehy, H., Dernoncourt, F., et al. “NoLiMa: Long-Context Evaluation Beyond Literal Matching.” ICML 2025; arXiv:2502.05167. Peer-reviewed. Minimal lexical overlap; 11 of 13 models below half baseline at 32K. https://arxiv.org/abs/2502.05167
• Hong, K., Troynikov, A., Huber, J. “Context Rot: How Increasing Input Tokens Impacts LLM Performance.” Chroma technical report, 14 July 2025. Non-peer-reviewed; open codebase. Distractor vs irrelevant; focused-vs-full gap; length isolated. https://research.trychroma.com/context-rot
• Anthropic. “Context windows.” Official documentation, verified September 2026. Window accounting; context rot named; overflow behaviour. https://platform.claude.com/docs/en/build-with-claude/context-windows
• Anthropic Applied AI team. “Effective context engineering for AI agents.” Published 29 September 2025. Finite attention budget; smallest high-signal set. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents