Context Rot: Why LLMs Fail Before Their Token Limit

A model sold with a 128,000-token context window does not reason reliably across 128,000 tokens. Adobe Research's NoLiMa long-context benchmark scored GPT-4o at a near-perfect 99.3% on short inputs, then watched it fall to 69.7% by 32K tokens, and most models it tested held 85% of their own baseline only out to a few thousand tokens. The window is a memory ceiling, not an accuracy guarantee. If you push long documents, chat history, or accumulated tool output into a prompt and trust the number on the spec sheet, this is the failure mode that quietly erodes your output.

What does context rot actually mean?

Context rot is the measured decline in an LLM's accuracy as its input grows longer, even when the task itself stays trivial. The term comes from a July 2025 technical report from Chroma by Kelly Hong, Anton Troynikov, and Jeff Huber, which evaluated 18 models and found that none of them use their context uniformly.

The distinction that matters for anyone specifying a system: a context window is the maximum number of tokens a model will accept without truncating. It says nothing about how well the model attends to token 90,000 versus token 900. Vendors advertise the former and let buyers assume the latter.

Chroma's cleanest demonstration is a needle-in-a-haystack task stripped to its simplest form, retrieving one planted fact from filler text. Performance still slid as the filler grew from a few hundred to several thousand tokens. When the task is that easy and accuracy still falls, the length itself is the variable doing the damage.

How far below the advertised window do models actually hold?

Far below it, and the gap is large enough to change architecture decisions. NoLiMa's authors define an "effective length" as the longest input at which a model still scores at least 85% of its base short-context accuracy, and for most models that number lands in the low thousands of tokens despite advertised windows of 128K or more.

The table below pulls the reported figures. Every model listed advertises a context window of at least 128K tokens, so the "effective length" column is the honest working limit, not the marketed one.

Eleven of the thirteen models NoLiMa scored retained half or less of their base performance by 32K tokens. GPT-4o degraded most gracefully and still shed nearly 30 points from its short-context peak. The advertised window, in other words, tells you where the model stops reading, not where it stops being right.

Why does more context lower accuracy at all?

Because attention has to spread across every token, and longer inputs raise the noise floor faster than they add signal. Transformer attention compares each token against every other, so a longer prompt does not make the relevant passage louder, it surrounds that passage with more plausible-looking distractors competing for the same attention budget.

NoLiMa isolates the mechanism precisely. When the question and the planted fact shared literal wording, models held near 98% accuracy at every tested length. Strip the lexical overlap so the model has to infer an association rather than pattern-match a phrase, and accuracy collapses as the context grows. Long-context retrieval leans on surface cues, and those cues get harder to trust the more text surrounds them.

Position compounds the problem. The 2023 Stanford study "Lost in the Middle" shuffled the answer-bearing document through a 20-document prompt and traced a U-shaped curve: models scored around 75% when the answer sat first, near 55% when it sat in the middle, and recovered toward the end. A fact buried at the center of a long prompt is the fact a model is least likely to use, and that pattern replicated across GPT-3.5, GPT-4, and Claude.

Does document quality or ordering change the outcome?

Yes, and in a direction that should unsettle anyone who assumes clean inputs help. Chroma ran its needle task over two haystack variants: one where the filler text read as a coherent, logically flowing passage, and one where the same sentences were randomly shuffled. All 18 models scored higher on the shuffled version.

The likely reason is that a coherent narrative gives the model a thread to follow, and it follows the thread instead of hunting for the planted fact. Structure that helps a human reader actively distracts the model. That finding undercuts a common instinct to feed models tidy, well-formatted prose and expect better recall.

Distractors make it worse in a predictable way. A single passage related to but distinct from the true answer measurably dropped Chroma's accuracy, and four distractors compounded the effect, with individual distractors landing unevenly. This is exactly the shape of a real RAG prompt, where a retriever returns several near-miss chunks alongside the correct one, so the benchmark's degradation is not an artifact of contrived inputs.

What breaks in real agent and RAG workloads?

The same rot, amplified, because agents and long-session assistants accumulate context by design. Chroma's LongMemEval results make the cost concrete: on conversational QA, prompts averaging roughly 113K tokens of full history trailed roughly 300-token focused prompts across every model family tested. Same question, same answer buried inside, and the only difference was how much surrounding history the model had to wade through.

Two failure modes show up beyond raw accuracy. Frontier models increasingly abstain on long, ambiguous inputs rather than commit, and Chroma flags Claude Opus 4 and Sonnet 4 as showing the most pronounced focused-versus-full gap largely because they decline to answer. On a repeated-words stress task, Opus 4 refused 2.89% of attempts and GPT-4.1 refused 2.55%, clustering around the 2,500-word mark. An agent that silently stops answering is a harder problem to debug than one that answers wrong.

For a long-running agent the implication is structural. Every tool call, retrieved document, and prior turn you leave in the window is not free working memory, it is added noise that lowers the odds the model uses the one detail that matters on this step. The context you carry forward has a quality cost, not just a token cost.

How should you architect around it?

Treat the context window as a budget to spend sparingly, not a buffer to fill. The through-line across all three studies is that fewer, more relevant, better-placed tokens beat more tokens, so the engineering goal is precision of input rather than volume.

Retrieval is the first lever. Instead of passing 50 chunks and hoping the model finds the right one, retrieve a wider set, then rerank and cut to the handful that actually answer the query, which is where a dedicated cross-encoder pass earns its cost. Tightening what your retriever emits before it reaches the model, for example with the approach in this walkthrough on adding contextual retrieval to a RAG pipeline, directly attacks the distractor problem NoLiMa and Chroma both surface.

For agents and long chats, manage the running context deliberately rather than letting it grow unbounded. Summarize or drop stale turns on a schedule so the live window stays close to the focused-prompt regime where models are reliable, a trade-off worked through in this piece on when to summarize versus truncate agent context. Where you can, place the decisive material near the start or end of the prompt rather than the middle, and validate at your own real input lengths instead of trusting the marketed window.

What are the trade-offs and what should you watch?

The uncomfortable trade-off is that the cheapest-looking option, dumping everything into a large window, is often the least accurate. Long-context prompting saves engineering effort over building retrieval and compaction, and prompt-caching economics make a big static prefix feel nearly free per call, but the accuracy tax it imposes does not show up on the invoice. It shows up as wrong answers your evals may not be sized to catch.

Watch two things. First, benchmark numbers age fast, and a model that fails NoLiMa today may be tuned to pass it next quarter without necessarily fixing the underlying attention behavior, so re-test rather than assume progress. Second, the advertised context window will keep climbing toward a million tokens and beyond, and that number will keep meaning less than it sounds, because a larger buffer the model reads unevenly is still a buffer the model reads unevenly.

The practical posture is skeptical by default. Chroma released its evaluation toolkit on GitHub, so before you commit an architecture to a given input length, run the test against your own model and your own documents. The right context length for your system is an empirical question, and the spec sheet is not the answer.

Sources

• Chroma, "Context Rot: How Increasing Input Tokens Impacts LLM Performance" (Hong, Troynikov, Huber, July 2025)
• Chroma, Context Rot evaluation toolkit (GitHub)
• Modarressi et al., "NoLiMa: Long-Context Evaluation Beyond Literal Matching" (Adobe Research)
• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (Stanford, 2023)