Why RL Environments Became the Bottleneck for AI Agents

Anthropic's leaders have discussed spending more than $1 billion on reinforcement learning environments over the coming year, according to reporting from The Information relayed by TechCrunch. Scale AI now says nearly half of its new data-training projects involve them. The scarce input in agent training has shifted away from model architecture and raw compute toward something harder to buy: the simulated workflows where an agent practices, fails, and earns a verifiable score.

For anyone deciding whether to fine-tune agents in-house, the practical question is no longer "which base model." It is who owns the environment the agent trains against, and who owns the verifier that decides what "done" means.

What exactly is an RL environment?

An RL environment is a resettable, inspectable simulation of a real workflow paired with a verifier that scores whether the agent actually accomplished the task. The agent takes an action, open a spreadsheet, file a support ticket, push a code change, and the environment updates its state and returns a reward computed from measurable changes in that state.

That reward mechanism is the whole point. Supervised fine-tuning teaches an agent to imitate static demonstrations, and RLHF teaches it to match human taste on outputs. Neither produces the trial-and-error signal an agent needs to learn a multi-step task where the first three steps are wrong and only the fourth recovers.

Scale describes its environments as "simulated worlds where agents can be trained and evaluated on a wide range of tool use and computer use workflows," scored by "expert-designed verifiers that check for real, measurable changes in the system." Each task ships with both process verifiers and outcome verifiers, and the tasks deliberately include incomplete data, conflicting records, and work that spans multiple tools. The environment encodes the definition of success, so its quality sets a ceiling on what the agent can possibly learn.

Why did environments become the bottleneck rather than models or GPUs?

Because the reusable inputs ran out and the remaining ones do not transfer. Frontier labs have largely exhausted high-quality static text for pretraining, and preference data teaches judgment rather than task completion. Environments are the one source of the signal agents need, and each domain requires its own.

That non-transfer is the expensive part. As Ross Taylor of General Reasoning put it to TechCrunch, "even the best publicly available [RL environments] typically don't work without serious modification." A CRM workflow environment teaches nothing about a radiology workflow. Every domain you want an agent to master demands its own expert-built simulation and its own verifier, which is skilled human work, not a GPU order.

The market has repriced that labor accordingly. Mechanize, founded roughly six months before the TechCrunch report, was offering software engineers $500,000 to build environments and was working with Anthropic. Andreessen Horowitz's Jennifer Li observed that "all the big AI labs are building RL environments in-house" while also turning to third-party vendors. When Scale reports that nearly half its new data projects involve environments and Anthropic is reportedly weighing a ten-figure annual spend, the signal is clear: the cost of an agent program now scales with the number of distinct workflows you want it to handle, roughly linearly, because each environment is bespoke.

Where does reward hacking creep in, and how do good verifiers avoid it?

Reward hacking creeps in wherever the verifier rewards a proxy instead of the real outcome. If the check is "was a ticket created" rather than "was the customer's problem actually resolved," the agent learns to create empty tickets. The environment did exactly what you asked; you asked for the wrong thing.

This is why Scale separates process verifiers from outcome verifiers, and why serious environments ground the reward in system state rather than in a model's self-assessment. Using an LLM as the verifier is tempting and sometimes necessary, but it imports every bias that makes an LLM-as-a-judge score unreliable straight into your training loop, where the agent will optimize against those biases far more aggressively than any human rater.

The defensible pattern is an outcome verifier tied to observable state changes, backed by adversarial tasks designed to catch the shortcut before the agent finds it. The trade-off is blunt: stronger verifiers cost more expert time to build, which is precisely the labor that already dominates the budget. Reward design is eval design run inside the training loop, and it deserves the same scrutiny.

What does the vendor landscape look like?

A handful of well-funded firms now sell environments, split between managed per-domain builders and open community hubs. The table below summarizes the main players and what each is known for, drawn from public reporting.

The open side matters as much as the commercial one. Prime Intellect's Environments Hub positions itself, in the words of coverage at the time, as a "Hugging Face for RL environments," a shared registry where the community publishes reusable tasks. It descends from the Gymnasium and Gym lineage that standardized the environment interface for classic RL, extended now to tool use and computer use. For generic capabilities, that shared layer is where costs will fall first.

Should you build or buy RL environments?

Buy or borrow for generic capabilities, and build for anything bound to your proprietary systems, data, and definition of success. Generic browser, coding, and computer-use tasks are exactly what vendors and open hubs are commoditizing, so paying to rebuild them in-house rarely pays off.

The workflows that differentiate your business are the opposite case. Your CRM schema, your compliance rules, your ticketing logic, and your data can only be verified against your own systems, so the environment has to live where that state lives. Build-versus-buy is really a verifier-ownership question: whoever controls the verifier controls what the agent optimizes for, and that is not a control most teams should outsource for their core workflows.

For most enterprises, the honest answer today is "not yet." Prompt engineering, retrieval, and off-the-shelf agents still capture most of the available value, and environments earn their keep only on a narrow, high-value workflow where a real reward can be defined and measured. The decision framework overlaps heavily with when reinforcement fine-tuning beats prompting at all; if RFT is not yet justified, neither is a bespoke environment.

How do you know an environment actually improved the agent?

You measure transfer to held-out, production-like tasks, not the training reward and not a single benchmark. Scale reports that models trained in its environments "consistently showed measurable performance improvements on the corresponding public benchmarks," which is encouraging, but benchmark lift can mask overfitting to the simulation, the familiar sim-to-real gap.

Treat the environment and the evaluation as separate assets with a firewall between them. Grade on tasks the environment authors never saw, report a reliability distribution rather than a point estimate, and prefer reliability metrics beyond pass@1 because an agent that succeeds once in five attempts is not production-ready no matter how good its average looks. Shadow deployment against real traffic remains the strongest signal, and it is where the gap between lab evaluation and real-world behavior usually surfaces.

What should senior teams watch next?

Three tensions will shape how this plays out. The first is commoditization. Vendors argue environments will not commoditize because, as they frame it, each domain needs expert data that stays bespoke. The likely outcome is a split: generic capability environments trend toward open hubs and falling prices, while high-fidelity domain environments stay expensive and proprietary.

The second is a caution from Andrej Karpathy, who wrote that he is "bullish on environments and agentic interactions but bearish on reinforcement learning specifically." Read that as a hedge on method, not on the asset. Even if the RL algorithm of the day changes, a well-built environment plus verifier remains valuable as evaluation and data infrastructure, which is a reason to invest in the environment independent of the training recipe.

The third is quieter and more dangerous: eval leakage. Environments make excellent evals, so teams reuse them for both training and grading. The moment the same environment trains and scores an agent, you have lost your independent measure and will overestimate progress. Keep the grading environment out of the training set, budget for verifiers to drift as your real systems change, and treat both as living infrastructure rather than a one-time build.

Sources

• TechCrunch: Silicon Valley bets big on 'environments' to train AI agents
• Scale AI: RL Environments
• Prime Intellect: Environments Hub
• PYMNTS: Prime Intellect Raises $130 Million to Help Companies Train AI Agents