Agent Context Compaction: When to Summarize vs Truncate

Every long-running agent eventually collides with its own transcript. Chroma's July 2025 Context Rot report tested 18 frontier models and found accuracy degrades non-uniformly as input grows, in some tasks losing a large share of it well before the documented window limit. That's the real reason context compaction, summarizing older turns so an agent can keep working, graduated this year into a native API after living as client-side glue code. The open question is now narrower: what to compact, when to trigger it, and whether to summarize at all or simply drop old tool output.

Key takeaways

Anthropic's compaction API (beta header compact-2026-01-12) summarizes a conversation server-side when input tokens cross a threshold, default 150,000 and minimum 50,000, and runs on the Claude API, Amazon Bedrock, Google Vertex, and Microsoft Foundry.
Chroma's Context Rot study showed a 200K-token model can lose meaningful accuracy by roughly 50K tokens of input, with the weakest recall in the middle of the context rather than at the ends.
An August 2025 arXiv study, "The Complexity Trap," found that masking old tool observations halves agent cost versus the raw loop while matching, and sometimes beating, LLM summarization on SWE-bench Verified.
Tool-call output, not the human or model conversation, is usually the largest single consumer of an agent's token budget.
Compaction is billed separately: the summarization pass appears in a usage.iterations array and is excluded from the top-level input and output token counts.
Durable facts belong in external memory, not the rolling window, because compaction is lossy and repeated passes compound what they drop.

What Does Context Compaction Actually Do?

Compaction replaces the oldest part of an agent's context with a concise summary, so the loop continues instead of hitting the window wall or paying to resend everything each turn. In the native Claude implementation, the server detects the trigger, generates a summary, writes a compaction block, and on later requests drops every content block before that summary. The agent keeps the thread of the task while shedding the raw material that produced it.

This differs from naive truncation, which lops off the oldest tokens with no attempt to preserve meaning. Truncation costs nothing but is blind: it will happily discard the decision an agent made 30 turns ago while keeping a verbose directory listing. Compaction spends a model call to keep the decision and drop the listing.

The trade is fidelity for tokens. A good summary preserves goals, constraints, and commitments. A bad one silently loses the one identifier the next tool call needed.

Why Does an Agent's Context Degrade Before It Fills Up?

Because models don't use long context uniformly, and quality falls long before the token count does. The Chroma team extended the classic needle-in-a-haystack test and reported that all 18 models they measured grew less reliable as input lengthened, following a U-shape: strong attention at the beginning and end of the context, weak in the middle.

Two findings matter for anyone running agents. First, degradation is neither gradual nor predictable; a 200K-token model can show serious accuracy loss around 50K tokens, so a window that looks a quarter full may already behave like one under strain. Second, distractors hurt more than empty space, and even coherent, well-structured filler pulls attention away from the relevant span.

The implication is that compaction is a quality lever, not only a capacity lever. You compact to keep the model sharp, not merely to fit under a limit. I go deeper on that in context engineering for long-horizon agents.

How Does Anthropic's Native Compaction API Work?

It runs server-side, keyed off input-token count, and requires the beta header compact-2026-01-12 plus a config object of type compact_20260112. You set a trigger; when input tokens reach it, Claude summarizes before answering. The default trigger sits at 150,000 input tokens and the floor is 50,000.

The parameters are few and worth knowing before you enable it.

Parameter	Default	What it controls
`trigger`	`input_tokens`, 150,000	Token count that fires a compaction pass; minimum 50,000
`instructions`	null	Custom summarization prompt; replaces the default one entirely
`pause_after_compaction`	false	Stops after summarizing so you can inspect or edit before continuing

Two operational details catch teams off guard. Summaries stream as a single delta rather than token by token, so a UI built for incremental text needs a special case. And compaction composes with prompt caching: put a cache_control breakpoint on the compaction block and a separate one at the end of the system prompt, so a compaction event doesn't invalidate the cached system context. That pairing matters if you already lean on prompt caching to cut API costs.

Should You Summarize or Just Drop Old Tool Output?

Often you should drop it. The August 2025 study "The Complexity Trap" ran five model configurations on SWE-bench Verified and found that masking old environment observations, hiding the bodies of earlier tool results, cut cost roughly in half against the raw agent while matching, and sometimes edging past, the solve rate of full LLM summarization. A hybrid of masking plus selective summarization saved a further 7%-11%.

The reason is where the tokens actually sit. In a tool-using loop, the transcript of what the agent said stays small; the bodies of what its tools returned, file dumps, search results, API payloads, dominate the budget. You rarely need the full body of a command you ran 20 steps ago, only the fact that you ran it and what it concluded.

That reframes the choice. Summarization is the expensive, high-fidelity option; masking and truncation are the cheap options that work more often than the summarize-everything reflex assumes.

Strategy	Relative cost	Fidelity	Best for
Raw loop, no management	Highest	Full, until context rot sets in	Short tasks that never approach the window
Truncation	Lowest	Lossy and blind	Chat where only recent turns matter
Observation masking	Low	Keeps actions, drops output bodies	Tool-heavy coding and ops agents
LLM summarization or compaction	Medium to high	High, but lossy across repeats	Research and planning where old reasoning matters
External memory	Low per turn	Durable and exact	Facts, IDs, and decisions that must survive

None of these is strictly best. Summarization earns its cost when the semantic content of old turns genuinely informs the next step, as in a long research or planning task. Masking wins when old tool output is bulky but disposable, which describes most coding and operations agents.

When Should Compaction Trigger?

Trigger on a threshold you choose, not at the moment you run out of room. A sensible default fires around 70% of the usable window, leaving headroom for the summarization pass itself and for the response that follows. The native API's floor of 50,000 tokens exists for the same reason: compacting a short context wastes a model call and can strip detail the agent still needs.

Fire too early and you pay for summaries you didn't need and discard detail prematurely. Fire too late and you eat both the cost of a bloated context on every turn and the accuracy loss from context rot. A June 2026 line of work on self-compacting agents argues the model itself should decide when to compact, and reports that a decision-based trigger can match a fixed threshold at a fraction of the token cost, though the approach is newer and harder to make predictable.

For most teams a fixed threshold with a sensible floor is the right starting point, instrumented so you can see how often it fires and what each pass costs.

What Should You Never Route Through Compaction?

Anything you cannot afford to lose. Compaction is lossy by construction, and its losses compound: a fact summarized away in pass one cannot be recovered in pass three. Identifiers, credentials, schema definitions, and the explicit decisions an agent has committed to belong in durable memory outside the rolling window, retrieved on demand rather than trusted to survive a summary.

This is the practical seam between compaction and memory. Compaction manages the transient working set; an external store holds the ground truth. The pattern I use in production keeps a structured record of decisions and entities and lets the window churn freely around it, which I cover in AI agent memory in production.

A useful test: if losing a piece of context would make the agent silently wrong rather than merely slower, it should never depend on a summary.

How Do You Budget and Bill for Compaction?

Account for the summarization pass separately, because the API does. Each compaction event is its own iteration with its own input and output tokens, surfaced in a usage.iterations array, and the top-level input_tokens and output_tokens deliberately exclude it. To know true spend, sum every entry in that array rather than reading the headline numbers.

The magnitudes are lopsided. In Anthropic's own documented example, a single compaction iteration reads about 180,000 input tokens and writes roughly 3,500 tokens of summary, after which normal turns run against the compact context. You pay a large one-time read to buy many cheaper subsequent turns, which is why a threshold that fires too often quietly inflates cost.

Prompt caching is the main lever to offset that read, and solve rate is the metric to protect. Track cost per completed task, not tokens per call, and measure it the way you would any agent reliability number, as I argue in measuring agent reliability beyond pass@1.

What Are the Trade-Offs, and What Should You Watch?

The central trade-off is fidelity against cost, and it doesn't resolve once. Every compaction pass is another lossy step, so an agent that compacts five times has summarized a summary of a summary, and small omissions accumulate into confident errors. Evaluate a compaction policy on completed-task quality before you ship it, not on how neatly it fits the window.

Three things deserve monitoring. Native compaction is convenient but couples your context strategy to one provider's summarizer and beta surface, so keep a client-side fallback if portability matters. Decision-based triggers look promising but forecast less cleanly than a fixed threshold. And because degradation hides until it doesn't, watch solve rate as context grows rather than trusting that a half-full window is a healthy one.

The quiet lesson from this year's research is that the expensive option isn't always the better one. Masking and eviction beat summarization more often than teams expect, and the right default is to reach for the cheapest strategy that preserves what the next step actually needs.