Tuning Reasoning Effort: When More Thinking Pays Off

Reasoning effort is the most expensive setting most teams never tune. It decides how many hidden tokens a model spends thinking before it answers, and those tokens are billed as output. Set it too high and you pay several times more for a point or two of accuracy you often can't even measure; set it too low on a genuinely hard task and the model rushes to a confident wrong answer. The skill isn't picking "high" and forgetting it. It's matching the dial to the work.

What does reasoning effort actually control?

It controls how many private reasoning tokens the model generates before it writes the answer you see. Those tokens never appear in the response body, but they occupy the context window and land on your invoice as output tokens. OpenAI's guidance is explicit: reasoning tokens "are not visible via the API" yet "are billed as output tokens," and you can read the exact count in the response's usage object under output_tokens_details.reasoning_tokens.

That single fact reframes the setting. Reasoning effort isn't a quality switch you flip on. It's a spend dial. At high or max, a model can emit tens of thousands of invisible tokens for a one-paragraph answer, and you pay output rates on every one of them.

The models also reason adaptively within a level. OpenAI notes that GPT-5-class models use "fewer tokens for simpler tasks and think harder for complex tasks," so the effort setting is a ceiling and a bias, not a fixed quota. The senior-engineer implication: your cost per request under a high setting is a distribution, not a number, and the tail can be brutal on adversarial or ambiguous inputs.

How much does higher effort really improve accuracy?

Less than the price increase implies, and past a point, nothing at all. Independent evaluations of GPT-5-class models repeatedly find that moving from medium to high effort buys a handful of accuracy points while multiplying token consumption by four or five times. The curve is concave: the first units of thinking pay well, the last ones barely register.

The frontier numbers make the cost concrete. Artificial Analysis measured GPT-5.6 Sol at max reasoning scoring 59 on its Intelligence Index at roughly $1.04 per task and about 15,000 output tokens per task as of July 9, 2026, and that variant is more token-efficient than most models at its intelligence tier. Efficient or not, max effort is where the meter runs fastest.

Coding is the clearest case of over-paying. Reporting on Anthropic's own benchmark data shows Claude Opus 5 performance on coding tasks peaking at medium effort and getting worse, not better, once you push past it. A model given room to second-guess a correct plan will sometimes talk itself out of it. If you default every coding call to max, you can pay triple for a lower pass rate.

When does more thinking stop helping, or start hurting?

When the bottleneck is knowledge the model doesn't have, extra reasoning can't manufacture it. A 2025 study, "Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yet," evaluated GPQA, AIME, and LiveCodeBench and found that longer reasoning budgets barely move accuracy on items that hinge on specific facts the model never learned. Thinking harder about a question you can't answer produces a longer wrong answer.

The failure has two flavors worth separating. On knowledge-bound tasks, more compute is wasted because the gap is informational, not computational; the fix is retrieval or fine-tuning, not a bigger budget. On tasks the model can do, over-thinking introduces a different risk: the model elaborates past the correct answer, second-guesses a sound step, or over-fits to a distractor in the prompt.

This is why "always max" is a defensible-sounding policy that quietly loses money and, on some task classes, accuracy. The instinct that more deliberation is safer comes from human work. It doesn't transfer cleanly to a system that pays by the token to deliberate and has no idea when it already had the answer.

How do the major providers expose the control?

Three vendors, three different knobs, same underlying trade. OpenAI ships a single enum, Google ships a level plus a legacy numeric budget, and Anthropic ships a hard token ceiling. The table shows how they line up.

The mechanics differ in a way that matters operationally. OpenAI's enum lets the model decide how much of the budget to use within a level, so you constrain the ceiling but not the actual spend. Anthropic's budget_tokens is a firmer cap: you name the maximum thinking tokens and the model works within it. Google's move from a raw thinkingBudget integer to a named thinking_level tracks the same lesson every team eventually learns, which is that a semantic level is easier to reason about than a magic number.

One caution when you compare invoices across vendors: reasoning tokens are priced at output rates everywhere, and output is the expensive side of the ledger. Claude Opus 5, for instance, lists at $5 per million input tokens and $25 per million output tokens, so a reasoning-heavy answer is charged almost entirely at the $25 rate.

How should you set a default reasoning effort per task class?

Match the dial to task difficulty, not to importance. A revenue-critical classification call is still a classification call, and it doesn't need to think. The most reliable policy sets a per-route default and lets only genuinely hard routes escalate.

Two refinements make this hold up in production. First, treat effort as a routing decision, the same way you already route between a small and a large model; the cheapest correct answer often comes from a smaller model at medium effort rather than a frontier model at max. Second, escalate on evidence, not on nerves: run the default, and only re-run at a higher effort when a verifier, a confidence check, or a failed tool call says the first pass was weak.

If you're already routing across models to control spend, reasoning effort is the second axis of the same grid, and the two compound. I've written more on the model side of that decision in LLM model routing and on when a smaller model is the right call in when to use small language models.

How do you cap runaway reasoning cost?

Put a hard ceiling on generated tokens and instrument what you actually spend. OpenAI's max_output_tokens limits the total the model can generate "including reasoning tokens, visible output tokens, and non-visible formatting tokens," which turns an open-ended reasoning budget into a bounded one. Anthropic's budget_tokens does the same job from the other direction by naming the thinking allowance up front.

Instrumentation is the half teams skip. Because reasoning tokens are invisible in the response, a request can cost 10x its neighbor with no sign in the output. Log output_tokens_details.reasoning_tokens (or the Gemini and Anthropic equivalents) per request, alert on the p95, and you'll find the small set of prompts that quietly drive most of the bill.

Research is starting to make the budget itself controllable rather than a blunt cap. "BudgetThinker" (2025) inserts special control tokens during inference to tell the model how much of its budget remains, then trains it with a length-aware reward so accuracy holds across a range of budgets. It's early, but it points at where this goes: models that spend deliberately against a target instead of thinking until they hit a wall. Until that ships broadly, a cap plus per-request logging is the practical control.

What should you watch as reasoning models evolve?

Watch three things: latency, variance, and benchmark drift. Latency is the one users feel. A max-effort answer can take many seconds of hidden thinking before the first visible token, which is fine for a batch job and unacceptable for an interactive assistant. For anything a person waits on, a lower effort that streams sooner often beats a higher one that scores marginally better on a benchmark nobody in the loop cares about.

Variance is the one finance feels. Because effort is a ceiling and the model spends adaptively, per-request cost is a distribution with a heavy tail. Budget against the p95, not the mean, and revisit it whenever you change models, because a new release can shift the whole curve.

Benchmark drift is the trap for decision-makers. Public leaderboards report models at max effort because that's where the top score lives, but max is rarely the configuration you should run. The number that matters is accuracy at the effort you can afford at your latency budget, and that's a measurement you have to make on your own tasks. Tie it to a reliability metric rather than a single pass, an argument I make in measuring AI agent reliability, and fold the reasoning-token line into the same discipline you apply to overall spend in AI token budgeting.

Sources

• OpenAI, Reasoning models guide (reasoning_effort values, token billing, max_output_tokens)
• Google, Gemini API thinking documentation (thinking_level, thinking-token pricing)
• Anthropic, Extended thinking documentation (budget_tokens)
• Artificial Analysis, GPT-5.6 benchmarks across intelligence, speed, and cost
• Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yet (arXiv)
• BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens (arXiv)