When to Use Batch LLM Inference Instead of Real-Time APIs

Any workload where no person waits for the answer is overpriced the moment you route it through a real-time API. Anthropic, OpenAI, Amazon, and Google all sell the identical models at 50% off through asynchronous batch endpoints, and that discount stacks with prompt caching. What you give up is a completion window measured in hours instead of milliseconds, plus a little orchestration you now own. This piece covers when that trade favors you, how the four providers actually differ, and what the savings look like on a job you would recognize.

Key takeaways

Anthropic, OpenAI, Amazon Bedrock, and Google Vertex each price asynchronous batch inference at 50% of their standard per-token rates for the same models, per each provider's own documentation.
The batch discount and prompt caching are independent discounts that compound. OpenAI's docs put cached input on a batched request as low as $0.625 per million tokens, a 75% cut from the $2.50 standard rate.
Batch trades latency for price. Providers quote a 24-hour completion window, though Anthropic reports most batches finish in under an hour.
Failed, canceled, and expired requests are not billed, so a batch that only partially completes costs you only for what it produced.
The right test is the clock, not the task. If a human is waiting on the response, batch is the wrong tool; if a queue is waiting, it usually is the right one.

What Does Batch Inference Change About Your Bill?

It halves the per-token price for the identical model and changes nothing about output quality. The weights, the context window, and the sampling are the same, so a batched Claude Opus 5 or GPT-5 response is the same kind of output you would get synchronously. The only thing you surrender is the immediate reply.

The discount exists because of how providers schedule the work. A batch job carries no latency guarantee, so it can be run against spare capacity and slotted around peak synchronous demand. Anthropic applies the 50% cut to both input and output tokens; OpenAI, Amazon Bedrock, and Google Vertex all land on the same headline figure for the models they support.

The implication is worth stating plainly, because teams get it wrong: batch is not a cheaper, weaker tier. It is the same model at half price with a slower delivery promise. Any cost model that treats batched output as lower quality is mispricing its own pipeline. If you already think in terms of the levers that move inference cost, batch is the single largest one available for offline work, and it requires no change to the prompt.

Which Workloads Belong in a Batch Queue, and Which Don't?

Any job whose result lands in a database, a report, or a downstream pipeline rather than in front of a waiting user belongs in a batch queue. The boundary is not how hard the task is; it is whether latency is part of the product.

Strong fits, and the ones that show up most often in practice:

Evaluation and regression suites run against thousands of test cases
Content moderation and classification over large volumes of user-generated data
Bulk record enrichment, tagging, and structured extraction from documents
Embedding backfills when you re-index a corpus or change models
Synthetic data generation for fine-tuning and eval sets
Overnight summarization, report generation, and scheduled analysis

The exclusions are just as clear: interactive chat, agent tool loops where a person watches each step, and anything sitting behind a latency SLA. The subtle case is a product that looks real-time but hides a batchable half, such as a nightly re-scoring pass or a periodic content refresh. Split those jobs out and the offline portion drops to half price while the interactive portion keeps its speed.

How Do the Major Batch APIs Compare?

All four providers reach the same 50% headline, but their request limits, windows, and result handling differ enough to matter once you operate at scale. The table below reflects each provider's current documentation.

Provider	Discount vs standard	Completion window	Per-batch limits	Stacks with caching	Result retention
Anthropic Message Batches	50% off input and output	Up to 24h; most under 1h	100,000 requests or 256 MB	Yes, best-effort cache hits	Results 29 days
OpenAI Batch API	50% off	Up to 24h, often faster	50,000 requests; 200 MB file; 2,000 batches/hour	Yes, automatic caching	Output deleted after 30 days
Amazon Bedrock batch	50% off on-demand, select models	Asynchronous, no strict SLA	Model and quota dependent	Model dependent	Written to your S3 bucket
Google Vertex batch	50% off input and output	Asynchronous	Dataset based (GCS or BigQuery)	Model dependent	Written to your storage

The differences that bite at volume are mundane but real. Anthropic caps a batch at 100,000 requests or 256 MB and keeps results for 29 days. OpenAI accepts up to 50,000 requests in a 200 MB file, runs the Batch API in a separate rate-limit pool from synchronous traffic, and deletes output files after 30 days. Bedrock and Vertex write results back to object storage you own, which changes your retention and access model rather than theirs.

Can You Stack Batch Pricing With Prompt Caching?

Yes, and this is where the economics get interesting: batch and caching are independent discounts that compound on the same request. Caching discounts the repeated prefix, such as a long system prompt, a few-shot block, or a shared document; batch discounts everything else.

OpenAI's prompt caching guide spells out the combined effect: cached input on a batched GPT-5.4 request can cost $0.625 per million tokens, a 75% reduction from the $2.50 standard input rate. Anthropic confirms the same stacking, with one caveat specific to how batches run.

Because batch requests process concurrently and out of order, Anthropic serves cache hits on a best-effort basis, and observed hit rates range from 30% to 98% depending on traffic shape. To push toward the high end, put identical cache_control blocks on every request in the batch and use the 1-hour cache duration, since a batch can easily outlive the default 5-minute cache window. The jobs that benefit most are high-prefix-reuse workloads, for example classification against a long shared rubric, where the same instructions ride in front of every record. If caching is new to your stack, start with the mechanics in how prompt caching cuts API cost before layering batch on top.

What Does the 24-Hour Window Actually Cost You Operationally?

The price break arrives with orchestration you now own: submission, polling, matching, and retry. None of it is hard, but pretending it is free is how batch pipelines break in production.

The shape of the work is consistent across providers. You submit a set of requests, each tagged with a custom_id you assign. You poll for status, since Anthropic reports a batch as in_progress and then ended. Results come back as JSONL and, critically, can return in any order, so you match them to inputs by that custom_id rather than by position.

Billing follows completion, not submission. Errored, canceled, and expired requests are not charged, which is generous, but it means your code has to detect them and resubmit. A batch that does not finish inside 24 hours expires and drops its unfinished requests, and Anthropic notes that processing can slow under high demand, pushing more requests toward that expiry edge.

So the sober planning assumption is a durable queue, idempotent resubmission keyed on custom_id, and a synchronous fallback for the tail if a hard deadline looms. Treat the batch pipeline as a system with retries and observability, the same way you would treat any asynchronous job runner, not as a single API call that happens to be cheaper.

How Much Does Batch Actually Save on a Real Workload?

Enough that on any large offline job it is the first optimization to reach for. Take a one-time enrichment of one million records on Claude Haiku 4.5, roughly 1,000 input tokens and 200 output tokens each.

At standard rates of $1.00 per million input tokens and $5.00 per million output tokens, that is 1,000 MTok of input at $1,000 plus 200 MTok of output at $1,000, for $2,000. Run the same job through the Message Batches API at Anthropic's batch rates of $0.50 and $2.50 per million, and it halves to $1,000. Add a shared, cached instruction block in front of every record and the input line falls further still.

The savings scale linearly with volume, which is exactly why they matter. Ten million records is a $20,000 versus $10,000 decision; a nightly job multiplies the gap by 30 every month. Batch pays off most precisely where your token spend is already largest, and it does so without touching model choice or prompt quality. Where volume and latency tolerance both vary across a product, batch pairs naturally with model routing, sending offline bulk to a cheaper model and a batch queue while interactive paths keep the faster tier.

What Should a Senior Team Watch For?

Three failure modes account for most of the disappointment: latency variance, capacity throttling, and misclassifying interactive work as batchable. Under provider load, completion can drift toward the 24-hour edge, so any process with a fixed downstream deadline needs a fallback path. Server-tool batches, such as those using web search, get throttled per organization, and extended-output jobs can take over an hour each, which stretches wall-clock time even when per-token cost stays flat.

The organizational risk is quieter and more damaging. A team under a cost mandate routes a user-facing feature to batch, ships a slower product to save money, and calls it an optimization. It is not one. Batch is free money on offline work and a broken promise on interactive work, and the line between them is the clock, not the ledger.

The migration that works is unglamorous: inventory every job with no human in the loop, move those to batch first, measure the drop, and only then look for the hidden offline halves inside real-time features. Keep a synchronous fallback for deadline-sensitive tails, and treat batch, caching, and routing as one cost system rather than competing tactics.