When Reinforcement Fine-Tuning an Agent Beats Prompting

Most teams reach for a better prompt when an agent fails a multi-step task. Past a point, prompting stops moving the number, and the next real lever is training the agent on its own tool-use trajectories against a verifier that scores whether the job actually got done. Reinforcement fine-tuning (RFT) is the technique labs now use to push tool-using agents past the ceiling that supervised data sets, and in 2026 it carries a clear economic threshold that decides whether it belongs in your stack.

When Should You Reinforcement-Fine-Tune an Agent Instead of Prompting It?

Only after prompt and tool redesign plateau and you can define success through a deterministic check: a passing test, valid JSON, a correct CLI command, a matching math answer, or a simulator outcome. If a qualified human still argues about whether an answer is right, the reward is too subjective to train on and you should stay with prompting and evals.

The practical hierarchy runs from cheapest to most expensive intervention. Each rung earns the next only when it stops paying off.

The line that matters for a budget owner: RFT sits at the bottom of that ladder for a reason. It's the answer when everything above it has been tried and the failures are verifiable, not the first thing you build. If you haven't yet built the evaluation harness that would define the reward, you're not ready to train. That harness is worth building either way, which is the quiet reason to start there. For the supervised rung below RFT, my walkthrough of LoRA and QLoRA fine-tuning covers the mechanics.

What Does RFT Actually Change Inside the Agent?

It optimizes the policy on entire trajectories, the full sequence of tool calls, their results, and the final outcome, rather than on isolated text completions. That's the difference between base RFT, which scores a single response, and agent RFT, which scores a whole run and lets the model discover tool-use strategies your demonstrations never showed.

The workhorse algorithm is GRPO. It samples several completions per prompt, scores each with the verifier, and computes an advantage for every response relative to its group:

A_i = (r_i - mean(r_1..G)) / std(r_1..G)

Because the group mean stands in for a learned value function, GRPO removes the separate critic network that PPO-style RLHF carries, cutting memory roughly in half. DeepSeek-R1 showed in January 2025 that this recipe, run without any supervised warm start, could reach competitive AIME, MATH, and coding scores, which is what pulled GRPO from research into production pipelines.

There's a well-documented reason RL earns its keep here rather than more SFT. Practitioner analyses summarize it as "SFT memorizes, RL generalizes": on held-out distributions, RL post-training recovered up to 99% of the out-of-distribution performance that heavy supervised fine-tuning had eroded on Qwen-2.5-7B, while keeping in-distribution accuracy. Supervised data teaches the pattern you showed; reward teaches the model to keep the pattern where you didn't.

How Do Verifiable Rewards Work for Tool-Using Agents?

A verifiable reward is a program that reads the agent's output and returns a score without human judgment, and the cleanest version is binary: +1 if the trajectory passes the check, 0 otherwise. DeepSeek-R1 trained on only two such signals, accuracy matching and format compliance, and that narrowness is a feature: a rule you can read is a rule the policy cannot easily game.

Three reward sources trade coverage against how gameable they are.

The middle row is where 2026 systems are converging. Meituan's LongCat reported a formal-reasoning generative reward model hitting 98.8% agreement with human labels, which let it reward Lean4 theorem proving without paying frontier-model rates on every rollout.

Reward shaping is where teams overreach. Dense intermediate rewards look helpful, for example crediting a coding agent for picking the right tool, producing a valid artifact, and passing a partial test on the way to the full one, but every added term is a new surface to hack. Real hacking patterns show up fast: gratuitous tool calls to farm a tool-use bonus, chains of thought padded to look like reasoning, and code that edits the test harness instead of solving the task. The defenses are diverse reward functions, sandboxed read-only test environments, and periodically refreshing any learned judge. When a reward can be satisfied without doing the work, the policy will find that path.

Why Does Multi-Step Credit Assignment Break?

Because a single outcome reward spread across a 20-call trajectory tells the model that the whole run was good or bad, not which step caused the result, and that signal degrades as trajectories get longer. This is the sparse-reward problem, and it's the hardest part of training agents rather than single-turn responders.

Three approaches sit on a spectrum. Trajectory-level rewards assign the outcome uniformly and are simplest to build but weakest on long runs. Process reward models score individual steps; folding turn-level rewards into GRPO has shown greater stability, faster convergence, and higher accuracy than trajectory-level baselines in published work. Outcome reward models plus light shaping keep a final-outcome signal but add partial credit for verified intermediate milestones.

GRPO itself fails in specific, nameable ways at scale, and knowing them is how you read a stalled training run.

DAPO packaged the fixes for the first two, adding token-level loss normalization, dynamic sampling, asymmetric clipping, and overlong reward shaping, and that combination is what let it beat an R1-Zero-equivalent on AIME with half the steps. If your team runs GRPO, budget for hitting at least one of these before anything trains cleanly.

What Does the Infrastructure Cost, and Why Does Async RL Matter?

A single RL run on a 70B model needs 32 to 64 H100 or H200 GPUs and lands around $10k to $50k in compute, and no serious program lands it in one run; plan for five to ten while you debug rewards and stability. Frontier programs run far past that. That price, not the algorithm, is what keeps RFT out of most stacks.

Synchronous training makes it worse. Agent trajectories run past 10,000 tokens with variable-length tool calls, so a synchronous loop leaves fast rollouts idling while slow ones finish, and reported GPU waste reaches 60% to 80%. Asynchronous RL decouples rollout from the gradient step so training proceeds as soon as enough on-policy data exists.

The scale gap is real. To make it concrete, NVIDIA reports post-training its Nemotron 3 Super across 21 verifier environments and 37 datasets, generating about 1.2 million rollouts. That's the volume of graded interaction serious RFT consumes, and it's why the compute bill dominates the decision.

What Do the Benchmarks Actually Show?

Where verifiers are precise, the gains are real and repeatable, but they cluster in domains with hard correctness signals, which is exactly where the method applies. Treat the headline numbers below as attributed lab results, not a guarantee for your workload.

For applied teams rather than labs, practitioner guides report a first agent-RFT pass adding roughly 10% to 25% task completion over SFT on held-out tasks when verifiers are precise, with curriculum and custom graders adding another 5% to 10% and cutting step counts. Those figures come from vendor and community write-ups, so verify them against your own held-out set before you bank the budget on them.

Can You Get RFT Without Running the Training Yourself?

Yes, through a managed API, but the current envelope is narrow. OpenAI's reinforcement fine-tuning runs only on the o4-mini reasoning model, and instead of a fixed answer key you supply a programmable grader that scores each sampled response so higher-scoring outputs get reinforced. It recommends starting with several dozen to a few hundred graded examples before committing to a large set, and it caps at 50,000 training and 1,000 test examples.

The eligibility bar is specific: the model has to already succeed sometimes, since RFT sharpens an existing capability and cannot bootstrap a skill from a 0% success rate, and the task has to be unguessable, so the reward reflects reasoning rather than luck. Medical diagnosis, legal research, and other expert-agreement domains are the examples OpenAI leads with.

If you're on API-hosted models and none of that fits, the higher-return work is upstream. Build the verifiable eval harness you would have used as the reward, structure your test suite as a difficulty curriculum from single-tool to long-horizon, and apply credit-assignment thinking to debugging by asking which tool call actually caused a failure. For narrow domains, a 7B to 14B model trained with a library-grade GRPO trainer runs in tens of GPU-hours on a single H100, a different budget than a 70B program. Deciding whether a smaller specialist is the right target at all is its own question, one I take up in when to use small language models in agentic systems.

What Should You Watch Before Committing?

Watch for a non-stationary environment first: if your tools, APIs, or policies change faster than you can regrade a dataset, the reward you trained against is already stale by deployment, and RFT will chase a target that has moved. Static, verifiable tasks are where it holds; live, shifting ones are where it quietly rots.

Watch the reward itself as the real deliverable. A precise verifier is harder to build than the training loop is to run, and a sloppy one teaches the model to game you. Keep the test environment sandboxed and read-only, keep a held-out set the training never sees, and refresh any learned judge on a schedule.

And keep the economics honest. For most teams on hosted models, RFT is not the right spend; evaluation infrastructure, reward design, prompt work, and retrieval return more per dollar until frontier models are demonstrably failing on your specific, verifiable domain. RFT earns its place when that condition is met and you can afford the debugging runs, not before. Measuring whether your agent is actually failing, and by how much, is the precondition; my note on measuring agent reliability beyond pass@1 is where I'd start that case.

Sources

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning (introduces GRPO), arXiv 2402.03300
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL, arXiv 2501.12948
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale, arXiv 2503.14476
• NVIDIA Technical Blog: Mastering Agentic Techniques, AI Agent Reinforcement Learning
• OpenAI: Reinforcement Fine-Tuning guide
• Zylos Research: RL Post-Training for Tool-Using Agents, GRPO, Async RL, and Reward Design
• verl: Volcano Engine Reinforcement Learning for LLMs
• OpenRLHF: An easy-to-use, high-performance RLHF framework