Context Engineering for Reliable Long-Horizon AI Agents

Long-horizon agents fail for a reason that has little to do with reasoning ability. As the run stretches across dozens or hundreds of turns, the context window fills with the agent's own history, and model accuracy starts to degrade well before that window is full. Context engineering is the discipline of deciding what stays in the window, what gets summarized, and what moves to external storage, and it now separates agents that hold up over long tasks from ones that quietly drift.

Key takeaways

Chroma's Context Rot study tested 18 models across Anthropic, OpenAI, Google, and Alibaba and found accuracy drops non-uniformly as input grows, even on simple retrieval, and that relevant text placed in the middle of a long context is recovered worst.
Anthropic reports that context editing alone improved agentic-search performance 29% over baseline, and pairing it with a memory tool reached 39%; in a 100-turn web-search evaluation, context editing cut token use 84% and let runs finish that would otherwise have exhausted the window.
The arXiv paper "Don't Break the Cache" measured prompt caching cutting API cost 45%-80% and improving time to first token 13%-31% across three providers over 500-plus agent sessions, while naive full-context caching sometimes raised latency.
Context editing and prompt caching pull against each other: removing tokens from inside a cached prefix invalidates the cache downstream and forces a re-prefill, so edits pay off only when timed at stable boundaries.
The operative lever is not a larger window but the smallest set of high-signal tokens, what Anthropic calls spending the model's attention budget deliberately.

Why do long-horizon agents get less reliable the longer they run?

Because model accuracy degrades as the input grows, not just when the window overflows. Chroma's Context Rot report evaluated 18 state-of-the-art models, including Claude Opus 4 and Sonnet 4, OpenAI o3 and GPT-4.1, Gemini 2.5 Pro, and Qwen3, and found performance varies significantly with input length even on tasks as simple as retrieval and copying text.

The degradation is not linear, which is what makes it dangerous to plan around. When the answer is semantically similar to the question, models hold up; when similarity is low, accuracy falls off faster as the context lengthens. A single distractor measurably reduces performance against a needle-only baseline, and some distractors hurt far more than others.

The most cited effect is positional. Models attend well to the start and end of a long input and poorly to the middle, the "lost in the middle" pattern first documented by Liu and colleagues in 2023. For an agent, every tool result, every retrieved document, and every prior step competes for that same limited attention, and the middle of a 100,000-token transcript is exactly where important state tends to sit by turn 60.

What is context engineering, and how is it different from prompt engineering?

Prompt engineering is about wording a single instruction well; context engineering is about curating the full set of tokens the model sees across many turns. Anthropic's engineering team frames it as managing system instructions, tools, external data, and message history together, and treats the transformer's attention as a budget that gets spent whether you plan it or not.

The guiding principle they state is to find the smallest possible set of high-signal tokens that make the desired outcome likely. That reframes the job. You are no longer trying to fit more into a large window; you are trying to keep the window lean enough that the model can still attend to what matters.

This is why "just use a model with a million-token window" solves less than it appears to. The window bounds what fits; context rot bounds what the model can reliably use. Those are different limits, and the second one binds first.

How much does managing the context window actually improve agent performance?

Enough to change whether long runs finish at all. On an internal agentic-search evaluation, Anthropic reports that context editing alone delivered a 29% improvement over baseline, and combining it with a memory tool reached 39%. The features shipped with Claude Sonnet 4.5 on September 29, 2025, in public beta on the Claude Developer Platform, Amazon Bedrock, and Google Cloud's Vertex AI.

Context editing automatically clears stale tool calls and results from inside the window as it approaches the token limit. The memory tool lets the model create, read, update, and delete files in a dedicated directory that persists across conversations, so state lives outside the window and gets pulled back in only when needed.

The reliability figure that matters most to anyone running agents at length: in a 100-turn web-search evaluation, context editing let agents complete workflows that would otherwise fail from context exhaustion while cutting token consumption 84%. That is a cost result and a completion-rate result at once, which is unusual and worth weighing against the engineering effort.

Which context-management techniques should you reach for, and when?

Four techniques cover most production needs, and they complement rather than compete. Compaction and context editing trim the live window; memory and note-taking move durable state outside it; sub-agents isolate work in clean windows; just-in-time retrieval loads data only when a step needs it.

Technique	What it does	Best when	Main trade-off
Compaction / context editing	Summarizes or clears stale turns as the window fills	Runs span many turns and old tool output is no longer needed	Summaries lose detail; edits can invalidate the prompt cache
Memory / structured note-taking	Persists state to files outside the window, retrieved on demand	State must survive across sessions or long gaps	Adds tool calls, latency, and a new surface to secure
Sub-agent architecture	Delegates focused work to agents with clean windows that return summaries	A task decomposes cleanly and parallelizes	Coordination overhead; a summary can drop context the parent needed
Just-in-time retrieval	Holds lightweight identifiers and loads full data at runtime	The corpus is large and only a slice is relevant per step	Retrieval latency per step; a bad query starves the model

In practice the strongest setups combine them. Anthropic notes that a hybrid of some upfront retrieval plus autonomous just-in-time exploration balances speed against relevance better than either extreme. The decision is rarely which one, and usually how to sequence them.

How does context management interact with prompt caching cost?

Directly, and not always in your favor, which is the part teams miss. Prompt caching reuses the key-value tensors for a shared prefix across requests, and the January 2026 arXiv study "Don't Break the Cache" measured it cutting API cost 45%-80% and improving time to first token 13%-31% across OpenAI, Anthropic, and Google over more than 500 agent sessions on DeepResearchBench.

Here is the tension. Context editing works by removing tokens from inside the running context. If those tokens sit inside a cached prefix, deleting them invalidates the cache for everything after the edit point, and the next call re-prefills that suffix at full price. The same paper found that naive full-context caching can paradoxically raise latency, while disciplined cache-block control gave more consistent gains.

The practical guidance follows from the mechanism. Keep the stable material, the system prompt and durable instructions, at the front where the cache holds. Place dynamic content at the end. Avoid interleaving volatile tool results into the cached region, and batch context edits to natural boundaries rather than trimming a token here and there every turn. Treat the cache prefix as an interface you try not to break, not a free optimization that always stacks.

How do you know your context strategy is actually working?

Measure completion and correctness on realistic long inputs, not synthetic needle tests, because the two diverge. Chroma's own results show a large gap between focused inputs of a few hundred tokens and full inputs above 100,000, and a clean needle-in-a-haystack score can hide that collapse.

Run the evaluation at the length and turn count your agent actually reaches in production, with real distractors present, and score whether the run finished as well as whether the final answer was right. A strategy that raises average quality but lowers completion rate on long runs is a regression for a long-horizon agent, and single-shot accuracy will not surface it.

This is the same reliability discipline that applies to agents generally: report the distribution across many trials, not one lucky pass. I've written separately on measuring agent reliability beyond the pass@1 number, and context strategy is one of the biggest levers on that distribution.

What are the trade-offs and open risks to watch?

Every context technique buys headroom by giving something up, and the failures are subtle rather than loud. Compaction and aggressive editing discard detail; a summary that drops an early constraint produces an agent that confidently violates it 40 turns later, with no error in the logs.

Memory tools add a persistent, writable store the model controls, which is genuine attack surface. Anything the agent can write, a prompt-injected instruction can potentially write too, so treat the memory directory with the isolation you would give any agent-controlled resource. Sub-agents shift the risk to the summary interface, where the parent inherits whatever the child chose to omit.

The cost picture also stays in tension. Moving state out of the window cuts per-call tokens, which lowers the bill and eases the pressure that the KV cache puts on inference cost, but each retrieval and memory call is another round trip that adds latency. The right balance depends on whether your workload is cost-bound or latency-bound, and it's worth deciding that explicitly before tuning. The teams getting this right treat context as an engineered budget rather than a container they fill until it breaks, the same mindset behind shrinking the token budget without shrinking the team.