Rate Limiting an MCP Server Against Agent Fan-Out

Put admission control inside the MCP server, keyed on the validated credential subject and the upstream API the tool hits, and shed excess calls as tool execution errors carrying a numeric retry hint. A single agent turn can open ten parallel tools/call requests against one API key, so a limiter sized for a human clicking a button will hand you a burst of upstream 429s and a model that retries straight back into the wall. The spec already assigns you this job: servers MUST "rate limit tool invocations" per the tools specification, and as of the 2026-07-28 revision there is no protocol session left to key that limit on.

Key takeaways

Enforce at two layers: an edge proxy keyed on the token subject, and an in process limiter sitting in front of each upstream client.
Key buckets on (tenant subject, upstream), never on the connection. Protocol level sessions were removed in 2026-07-28, so connection identity is not a quota identity.
Wait one or two seconds at most, then shed with isError: true and a numeric wait in the text, because clients feed tool execution errors back to the model for self correction.
Retry at most once inside a tool. The agent loop and the client are both already retrying you.
Treat an upstream 429 as a rate signal, not only a failure: cut your bucket rate and honor Retry-After instead of guessing a backoff.
Alert on shed ratio per tenant. A global error count hides the one account that is eating everyone else's quota.

Why agent fan-out overruns limits built for human traffic

Agent traffic differs from human traffic in concurrency shape, not in volume. A person using a dashboard issues one request, reads the result, and issues another. A model that has been handed a list tool and a detail tool will call the list tool once and then issue N detail calls in the same assistant turn, all of them in flight at the same moment, all of them carrying the same credential. Add a multi agent harness where three subagents share one MCP server and one upstream key, and your arrival process has burst factors a leaky bucket sized on average QPS never sees.

The second difference is what happens after you reject. A browser backs off. A model reads your error text, decides the call was worth making, and calls again immediately, often in parallel with the sibling calls that are also being rejected. If your error text says "try again later" the retry arrives in about 400ms. Rejection without a usable wait hint converts a quota problem into a retry storm that you built yourself.

Before tuning anything, measure the shape. Log one line per tools/call with the token subject, tool name, and arrival timestamp, then look at calls per subject in 1 second windows rather than per minute. Most teams find their p99 one second concurrency is five to twenty times their average, which is the number the limiter actually has to survive.

Where the limiter belongs in an MCP deployment

Use two layers with different jobs: an edge limiter that protects the server process, and an in process limiter that protects each upstream dependency. They are not redundant, because they count different things.

The edge layer can now work without parsing JSON-RPC bodies. The Streamable HTTP transport mirrors selected body fields into HTTP headers (Mcp-Method on every request, Mcp-Name carrying the tool name on tools/call) precisely so that load balancers and gateways can route and inspect without reading the body. The same page carries a warning worth obeying: intermediaries enforcing policy on mirrored headers, including "rate-limiting by tenant", SHOULD verify the MCP-Protocol-Version header indicates a revision that requires header and body validation, and reject the request otherwise. Older clients can send a header that disagrees with the body, and a gateway that trusts it will limit the wrong tool.

The edge layer cannot see amplification, which is why the second layer exists. One search_orders call may issue four upstream HTTP requests, one sync_customers call may issue two hundred. Only the server knows the multiplier, so the bucket that matters sits in front of the upstream client, not in front of the MCP endpoint. Do not push this responsibility onto the MCP client either. You do not control which harness connects, and a client side limit is advisory at best.

What to key the buckets on now that sessions are gone

Key on the tuple (authenticated subject, upstream), with tool class as an optional third dimension for expensive endpoints. The 2026-07-28 revision removed protocol level sessions and the standalone GET stream from Streamable HTTP, and instructs servers that receive an Mcp-Session-Id header to ignore it and to not mint or echo session IDs. Any limiter keyed on connection identity, session ID, or SSE stream is keyed on something the protocol no longer guarantees. If you are still carrying session shaped state, that migration is its own piece of work, covered in where MCP server state lives without sessions.

The subject must come from a token your server validated for its own audience, not from whatever bearer string arrived. Passing a client token through to the upstream also destroys your ability to attribute usage, since every tenant then looks like whoever the token belonged to. The failure modes are covered in why your MCP server must not pass tokens through.

One nuance that helps here: the tool set a server returns MAY vary by the authorization presented on the request, since credentials are per request input rather than connection state. That means the same subject that defines the visible tool list can define the bucket, and both stay correct when a client rotates credentials mid conversation.

How to shed load so the agent backs off instead of hammering

Shed as a tool execution error with isError: true, never as a JSON-RPC protocol error. The spec draws the line explicitly: protocol errors indicate problems with the request structure "that models are less likely to be able to fix", while tool execution errors "contain actionable feedback that language models can use to self-correct and retry with adjusted parameters", and clients SHOULD provide tool execution errors to the model. A -32603 tends to surface to the user as a broken server; an isError result surfaces to the model as an instruction.

The text is the interface. Give the wait in seconds as a number, say the arguments are fine, and say what not to do:

{
  "jsonrpc": "2.0",
  "id": 7,
  "result": {
    "resultType": "complete",
    "content": [{
      "type": "text",
      "text": "Rate limited by the orders-api quota for this account. Wait 4 seconds, then retry this call with the same arguments. Other tools share the same quota and will also fail."
    }],
    "structuredContent": { "error": "rate_limited", "retry_after_ms": 4000, "scope": "tenant" },
    "isError": true
  }
}

The structuredContent field is for the harness, which can sleep on retry_after_ms without asking the model anything. The text is for the model when the harness ignores it. Wording that makes an agent stop rather than thrash is a broader topic, treated in returning MCP tool errors agents can recover from.

Here is the admission path itself. A reservation style token bucket plus a concurrency semaphore, with a hard ceiling on how long a call may sit before it is rejected:

import asyncio, time

class Shed(Exception):
    def __init__(self, retry_after_s):
        self.retry_after_s = retry_after_s

class Admission:
    # One instance per (subject, upstream). Waits briefly, then sheds
    # rather than queueing behind an unbounded backlog.
    def __init__(self, rate, burst, concurrency, max_wait=2.0):
        self.rate, self.burst = rate, burst
        self.tokens, self.updated = burst, time.monotonic()
        self.sem = asyncio.Semaphore(concurrency)
        self.max_wait = max_wait

    def _reserve(self):
        now = time.monotonic()
        self.tokens = min(self.burst, self.tokens + (now - self.updated) * self.rate)
        self.updated = now
        self.tokens -= 1                 # may go negative: this call reserved future capacity
        return max(0.0, -self.tokens / self.rate)

    async def acquire(self):
        wait = self._reserve()
        if wait > self.max_wait:
            self.tokens += 1             # give the reservation back, nobody will use it
            raise Shed(round(wait, 1))
        if wait:
            await asyncio.sleep(wait)    # a client disconnect cancels this task right here
        try:
            await asyncio.wait_for(self.sem.acquire(), timeout=self.max_wait)
        except asyncio.TimeoutError:
            raise Shed(self.max_wait)

    def release(self):
        self.sem.release()

Wiring it into a tool is mechanical. Note that the Python SDK converts a raised exception into a tool execution error with isError set, though some versions replace the message with a generic string, so check what your version actually puts in the text before relying on the model reading your number:

@mcp.tool()
async def search_orders(query: str, ctx: Context):
    limiter = limiter_for(subject=auth_subject(ctx), upstream="orders-api")
    try:
        await limiter.acquire()
    except Shed as s:
        raise ToolExecutionError(
            f"Rate limited by the orders-api quota for this account. "
            f"Wait {s.retry_after_s} seconds, then retry this call with the "
            f"same arguments. Other tools share the same quota."
        )
    try:
        return await orders_api.search(query)
    finally:
        limiter.release()

How long to hold a call before handing back a task handle

Hold for one to two seconds, then either shed or hand back a durable handle, because the client is running its own timeout against you. The cancellation rules say implementations SHOULD set timeouts on all sent requests, MAY reset the clock on a progress notification, and SHOULD always enforce a maximum regardless. A call parked in your queue for ninety seconds gets cut by the client, and on Streamable HTTP that disconnect is itself the cancellation signal, which the server MUST treat as cancellation of the request.

That last point has a quota consequence people miss. If a queued call reaches the front of the line after its client vanished, executing it spends upstream quota on a result nobody will read. Check for cancellation immediately after acquiring the token, not only before the wait. In asyncio the cancellation propagates through the await asyncio.sleep(wait) above for free, which is one reason to sleep rather than spin. The wider class of orphaned work is covered in what happens when an agent cancels and your tool keeps running.

When the work is genuinely long rather than merely queued (a bulk export, a report that takes four minutes), stop holding the connection and return a task. The Tasks extension defines a CreateTaskResult with resultType: "task", carrying a taskId, a ttlMs, and a suggested pollIntervalMs, with the client polling tasks/get until a terminal status. That pollIntervalMs is a rate control lever in disguise: it is you telling the client how often to come back. Verify the client declared the extension in its per request capabilities before returning a task, since a client that did not opt in will not know what to do with the shape. For work that finishes in seconds but benefits from visible progress, progress notifications remain the lighter option.

How to react to upstream 429s without amplifying them

Treat the upstream 429 as an instruction to reprice your own bucket, not as a transient blip to retry past. When the response carries Retry-After, honor it: the header accepts either a delay in seconds or an HTTP date, per RFC 9110, and it is more accurate than any backoff curve you invent. Feed that value into two places at once, the error you return to the agent and the refill rate of the bucket for that subject, so the next twenty queued calls are shed cheaply instead of each discovering the 429 independently.

Retry at most once inside the tool, with full jitter on the sleep. Synchronized retries across a fan-out are the classic amplification case that the AWS Builders' Library piece on timeouts, retries and backoff with jitter describes, and MCP makes it worse because the layers stack: the upstream SDK retries, your tool retries, the agent retries, the harness retries. Four layers of three attempts is eighty one requests for one logical call. Pick one layer to own retries and make the others fail fast.

For sustained pressure, adaptive throttling beats a fixed ceiling. The client side throttling approach in Google's SRE book chapter on handling overload rejects locally in proportion to the observed reject rate, which keeps a struggling upstream from spending its remaining capacity on requests it will refuse anyway. Share that state across your worker processes (Redis or a shared counter), otherwise eight workers each learn about the 429 separately and you send eight times the probe traffic. Tools that mutate state need the retry story settled first, which is the subject of making MCP tools idempotent for retrying agents, and multi tenant quota pooling has the same shape described in why one API key throttles six accounts.

What to measure so you can tell shedding from breaking

Instrument four counters per (subject, tool, upstream) and one histogram: admitted, shed at the bucket, shed at the concurrency cap, upstream 429s, and time spent waiting in admission. The combination tells you which knob is wrong. High shed with zero upstream 429s means your bucket is tighter than the upstream limit and you are rejecting capacity you own. High shed and high 429s together means the bucket is right but the key is wrong, usually because one tenant's fan-out is draining a pool the others share. Near zero shed with rising upstream 429s means the limiter is bypassed somewhere, typically a code path that builds its own HTTP client.

Put the subject and tool name on the span, not only in a log line, so a slow agent turn can be traced to admission waits rather than to model latency. The attribute names are converging on the OpenTelemetry GenAI conventions, which is worth following so your MCP spans line up with the harness spans above them. Alert on shed ratio per tenant over a five minute window. A global error rate stays flat while one account is being starved, which is exactly the incident you want to catch early.

Trade-offs and what I would ship

The genuine choice is between shedding fast and queueing durably, and it turns on how long the work takes relative to a client timeout. Shedding fast keeps the server's memory bounded, keeps client timeouts from firing, and gives the model an explicit instruction it can act on. The cost is that a burst of legitimate work turns into a burst of retry round trips, each one costing tokens for the error text and the retried call, which is not free when the tool result is large. Queueing durably with the Tasks extension costs you a polling loop, a store that survives restarts, and client support you have to verify per harness, and it wins decisively for anything over roughly ten seconds.

What I would ship for a typical remote MCP server fronting a rate limited API: a per (tenant, upstream) reservation bucket sized at about seventy percent of the upstream published limit, a concurrency cap of eight to sixteen per tenant, a two second maximum admission wait, shed as isError with a numeric wait, exactly one jittered retry inside the tool for idempotent reads and zero for writes, and adaptive rate cuts driven by upstream Retry-After. Tasks only for the handful of tools that are genuinely long running. Edge limits keyed on the token subject using the mirrored Mcp-Name header, with the protocol version check the spec asks for.

Two failure modes to design against explicitly. Edge only limiting looks fine on dashboards while one tool amplifies ten to one into an upstream you do not control, which is the case that pages you at 3am. In process only limiting protects the upstream while a single hot tenant opens thousands of concurrent POSTs and exhausts your file descriptors before any bucket is consulted. Neither layer subsumes the other, and the cheapest version of both is still cheaper than the incident.