Measuring AI Agent Reliability: Beyond the pass@1 Number

The number on the benchmark card is almost always a capability score, not a reliability score, and the two diverge sharply for AI agents. An agent that resolves a task 61 percent of the time on a single attempt can complete that same task on eight consecutive attempts less than 25 percent of the time. If you are deciding whether to put an agent in front of customers or inside a revenue workflow, the metric that matters is not whether it can succeed but whether it succeeds every time a user expects it to. This article explains the metrics that expose that gap and how to use them before you ship.

Key takeaways

pass@1 reports average per-attempt success; pass^k reports the probability of succeeding on all k attempts. For any success rate below 100 percent, pass^k falls fast as k grows, and it is the honest proxy for user-facing reliability.
On the tau-bench tool-agent benchmark from Sierra, state-of-the-art function-calling agents solve fewer than 50 percent of tasks and score pass^8 below 25 percent in the retail domain, showing that consistency, not raw capability, is the binding constraint.
Agent success decays roughly exponentially with task length. A 2025 analysis of METR data fits a near-constant per-step failure rate, meaning each additional dependent step chips away at the odds the whole task completes.
METR finds the task length frontier models complete at 50 percent reliability has doubled about every seven months since 2019, but the 80 percent-reliability horizon is far shorter, so reliability lags capability by a wide margin.
Reliability responds to architecture, not only to bigger models: verification steps, task decomposition, lower sampling temperature, explicit reasoning, and human escalation move pass^k more cheaply than retraining.

What is the difference between pass@1, pass@k, and pass^k?

They answer three different questions about the same agent. pass@1 is the average success rate of a single attempt: run the task once, did it work. pass@k is the probability that at least one of k independent attempts succeeds. pass^k, pronounced "pass power k," is the probability that all k independent attempts succeed. The formulas make the divergence concrete. With a raw single-attempt success rate of c/n, pass^k is (c/n)^k, while pass@k approaches 1 as k grows for any agent with non-zero capability.

Philipp Schmid's worked example makes the stakes obvious. Take a flight-rebooking agent that succeeds 70 percent of the time on a single request. Over three attempts, pass@3 is roughly 97 percent, because at least one of the three will very likely land. But pass^3, the chance it handles three consecutive customer requests correctly, is only 34.3 percent. The same agent looks excellent or unacceptable depending entirely on which metric you quote. This is why pass@k is described as "exponentially forgiving": as k rises, almost any agent that is not hopeless will eventually stumble into a correct answer, which is useful for measuring capability during research but actively misleading for production planning.

The practical rule follows directly. Use pass@k in development to separate a capability gap from a consistency gap. If pass@k is low, the agent genuinely cannot solve the problem and you need a better model or better tools. If pass@k is high but pass^k is low, the agent can solve it but will not do so reliably, and the fix is engineering, not a larger model.

Metric	Question it answers	Formula	Use it for
pass@1	Will a single run succeed on average?	`c/n`	Headline capability, expected single-shot performance
pass@k	Will at least one of k runs succeed?	`1 - C(n-c, k)/C(n, k)`	Ceiling of capability, separating cannot-solve from inconsistent
pass^k	Will all k runs succeed?	`(c/n)^k`	User-facing reliability, escalation and staffing plans

Why does a 61 percent success rate collapse to 25 percent reliability?

Because independent-looking attempts multiply, not average. The clearest published evidence comes from tau-bench, the tool-agent-user benchmark introduced by Sierra, which measures agents in realistic retail and airline customer-service domains where the agent must follow domain rules and use tools against a database. tau-bench introduced pass^k specifically "to evaluate the reliability of agent behavior over multiple trials." Its headline result: state-of-the-art function-calling agents, including GPT-4o, succeed on fewer than 50 percent of tasks, and pass^8 sits below 25 percent in retail. In plain terms, ask the same well-specified task eight times and the odds it succeeds on all eight are worse than one in four.

The authors' conclusion is not that the models are weak. It is that they are inconsistent: they can follow the rules, but not every time, and the failures are not the same failures twice. That distinction is what separates a demo from a deployment. A retail refund agent that works in the pitch meeting and fails one time in three across a week of live traffic is not a smaller version of a working system; it is a different system that happens to share a screenshot. Understanding how agents chain tool calls and hold state is central here, which is why the discipline of agentic orchestration matters as much as model selection.

Why do longer agent tasks fail faster than their length suggests?

Because failure probability compounds across dependent steps, and the compounding is roughly exponential in the number of steps. A 2025 analysis asking whether AI agents have a "half-life" for success (arXiv 2505.05115) fits METR's data with a model in which the agent has a near-constant rate of failing during each unit of human-equivalent time. Under that model, the probability the whole task succeeds declines exponentially with task length, because a long task is a chain of subtasks where failing any one fails the whole. An agent can then be characterized by its own half-life: the human-task duration at which its success rate falls to 50 percent.

Recent work sharpens the mechanism and, usefully, points at the lever. "The Illusion of Diminishing Returns" (arXiv 2509.09677) shows that per-step accuracy itself degrades as the number of steps grows, and identifies a specific culprit the authors call self-conditioning: models become more likely to make mistakes when their context already contains their own earlier errors. The error rate is not just constant, it can accelerate once the transcript is polluted. The encouraging counterpart is that marginal gains in single-step accuracy "compound into exponential improvements in the length of tasks a model can successfully complete." A model that is only slightly more reliable per step can execute a dramatically longer task, which is why small reliability improvements are worth pursuing even when headline capability looks flat.

Two engineering implications follow. First, coherence problems compound with tool-call depth, so agents that hold long transcripts of their own intermediate work degrade in ways that are invisible to a single-turn benchmark. Second, because the transcript is the failure vector, disciplined context and token budgeting is a reliability intervention, not only a cost one: trimming stale tool output and prior errors from the working context directly reduces self-conditioning.

How fast is agent reliability actually improving?

Capability is improving quickly; reliability is improving far more slowly, and the two should not be conflated. METR measures a "time horizon," the length of task (by how long a human expert takes) that a model can complete at a given success rate. Its widely cited finding is that the 50 percent-reliability time horizon of frontier models has doubled roughly every seven months from 2019 through 2025, a trend so steady it has been called a Moore's Law for agents.

The detail that matters for deployment is the gap between the 50 percent and 80 percent horizons. The 80 percent-reliability horizon, the task length a model finishes eight times out of ten, is much shorter than the 50 percent horizon. A model that can handle a two-hour task half the time might only handle a fifteen-minute task with the four-out-of-five consistency a production workflow actually needs. METR also notes that measurements above 16 hours are unreliable with its current task suite, a reminder that the longest-horizon claims rest on the thinnest data. The lesson for a decision-maker: the impressive number in the announcement is almost always the 50 percent horizon, and your production bar is closer to the 80 or 95 percent horizon, which is a fraction of the length. This is the concrete shape of the lab-to-production gap that senior leaders keep encountering, a theme running through recent enterprise AI warnings.

How should you measure reliability before you ship an agent?

Run every evaluation task multiple times and report pass^k alongside pass@1, then let the spread drive decisions. A single-run evaluation is not an evaluation of an agent; it is a lucky or unlucky anecdote. Concretely:

Report the triple. For each configuration, publish pass@1 (expected single-run performance), pass@k (capability ceiling), and pass^k at a k that reflects real usage. If pass@1 sits close to pass^k, the agent is consistent. A wide spread is your reliability debt, quantified.
Pick k from the workflow, not the paper. If a customer session involves five sequential agent actions that all must succeed, k is 5. If an agent runs a task a hundred times a day and one silent failure is a real incident, evaluate at a k that reflects that exposure. pass^8 is a convention, not a law.
Turn pass^k into a staffing number. pass^k directly estimates how often a human will need to intervene. If pass^k over a session is 0.7, roughly three sessions in ten need a fallback, which is a concrete input to escalation design and headcount, not an abstract quality score.
Calibrate the judge. If an LLM grades the runs, calibrate it against a human-labeled gold set before trusting it, because an uncalibrated judge manufactures whichever reliability number you were hoping to see.
Gate on real scores and grow the suite from production. Put the pass^k threshold in continuous integration so a regression blocks a release, and feed failing production traces back into the evaluation set so the suite tracks how the agent actually fails in the field.

What architecture choices move reliability rather than capability?

Once pass@k confirms the agent can solve the task, reliability is an engineering problem with cheaper levers than retraining. The most effective, in rough order of leverage:

Verification and retry. Add a step that checks the agent's output against the goal and re-runs on failure. This converts a pass@k advantage into realized reliability, since you are explicitly buying more attempts where at least one must land.
Decomposition. Break a long task into shorter, independently verifiable subtasks. Because failure compounds with length, three verified five-step tasks are far more reliable than one unverified fifteen-step task, even at the same per-step accuracy.
Lower temperature and tighter instructions. Much inconsistency is sampling variance and ambiguous rules. Reducing temperature and removing instruction ambiguity narrows the spread between pass@1 and pass^k at no model cost.
Explicit reasoning. The self-conditioning research finds that letting the model think mitigates degradation from its own prior errors and extends the length of task it can execute in a single turn. Reasoning-heavy configurations trade latency and tokens for consistency.
Context hygiene. Prune stale and erroneous intermediate output from the working context to limit self-conditioning, and manage tool schemas carefully, since standards like the Model Context Protocol shape how much tool metadata competes for the same context.
Human-in-the-loop by design. When pass^k plateaus below the bar, route the residual to a person deliberately rather than pretending the gap does not exist. Designed escalation is a feature; silent failure is an incident.

What are the trade-offs and what should you watch?

Every reliability lever costs latency, tokens, or engineering time, and pass^k is demanding enough that chasing it to 1.0 is usually the wrong goal. Verification and reasoning add tokens and wall-clock time; decomposition adds orchestration complexity and more points of integration failure. The right target is the reliability the workflow actually requires, not the highest number achievable. A back-office summarization agent and a payments agent do not need the same pass^k, and spending equally on both wastes budget on one and under-protects the other.

Three things to watch. First, benchmark inflation: treat any single headline score, including a strong pass@1, as capability marketing until you see multi-run consistency. Second, the reliability-capability divergence: METR's data shows raw capability can climb while reliability barely moves, so a newer, smarter model is not automatically a more reliable one for your task, and it must be re-measured. Third, evaluation drift: an agent that passed at launch degrades as the model provider updates weights, as your prompts accrete, and as real inputs drift from your test set, which is why the pass^k gate belongs in continuous integration and the test suite must keep ingesting production failures. Reliability is not a launch milestone; it is a number you defend continuously.