Agentic & tool-use RL

Multi-turn trajectories and the 2026 frontier

Papers: The Landscape of Agentic Reinforcement Learning for LLMs: A Survey — Zhang et al., 2025 · Search-R1, ReTool, ToRL / Tool-Star (2025)

Some tasks require a model to act on the world before it can answer. They search the web, run code, read the result, and decide what to do next, looping until the task is done. Training that behavior requires assigning credit across a sequence of decisions, including decisions whose consequences appear much later.

From answer to trajectory

Agentic RLagentic RLReinforcement learning over multi-step, tool-using trajectories — the model acts, observes results, and acts again — rather than producing a single response. The 2025–26 frontier.See in glossary → reframes the model from a text generator into a policy acting in an environment. Instead of producing one completion, the model produces a trajectorytrajectoryThe sequence of states and actions in a rollout. For text generation, the tokens generated one after another, each conditioned on those before it.See in glossary →: a sequence of interleaved actions and observations. It writes a search query (action), receives results (observation), reasons over them, writes another query or calls a tool, observes again, and eventually emits a final answer. The reward usually lands only at the end (was the final answer correct?), and the training problem is to push the whole sequence of decisions toward trajectories that succeed.

Tool-use RLtool-use RLTraining a model with RL to call external tools (search, code execution, calculators) effectively, rewarding trajectories that use tools to reach correct outcomes.See in glossary → is the most concrete instance: the model’s action space includes calling external tools (a calculator, a code interpreter, a retriever, an API), and the environment returns their outputs as observations the model must then incorporate. This is the engine behind a wave of 2025 systems:

• Search-R1 trains the model to interleave reasoning with live search queries, learning when to look something up and how to use what it finds: RL over a retrieval-augmented trajectory.
• ReTool does the same for a code interpreter: the model learns to write code, run it, read the output (including errors), and use the result to reach the answer.
• ToRL / Tool-Star generalize this to multiple tools, with the model learning which tool to reach for and how to chain calls.

Multi-turn RLmulti-turn RLRL where an episode spans many interaction turns (with a user or an environment), requiring credit assignment across turns rather than within one response.See in glossary → is the umbrella: optimizing over many rounds of model–environment interaction rather than a single turn. And once there are many turns, a new question appears: how do you assign credit to individual turns? A turn-level rewardturn-level rewardA reward assigned to individual turns or tool calls within a multi-turn trajectory, giving denser feedback than a single end-of-episode reward.See in glossary → scores intermediate steps (was this search query useful? did this tool call move things forward?) rather than waiting for the final outcome, giving the optimizer a denser signal to work with.

Why longer interactions are difficult to train

The reasoning-RL machinery from earlier chapters (GRPO, verifiable rewards, group baselines) mostly carries over. What breaks is everything around the reward.

• Credit assignment across turns. If a 20-step agent fails, which step was the mistake? The bad answer might trace to a search query made fifteen steps earlier. AdvantageadvantageHow much better an action was than the baseline expectation: A = reward − value. Positive advantage pushes an action’s probability up, negative pushes it down.See in glossary → estimation over long horizons is exactly the regime where the group-mean baselinebaselineA reference value subtracted from the reward to reduce gradient variance without adding bias. Can be a learned critic, a group mean (GRPO), or a leave-one-out average (RLOO).See in glossary → is weakest and a trained value functionvalue functionThe expected return from a given state under the current policy. A learned value function (the critic) provides a baseline that reduces the variance of policy-gradient updates.See in glossary → (the VAPO impulse) starts to look attractive again.
• Sparse, long-horizon rewards. A single binary “did the task succeed” signal at the end of a long trajectory is a very thin gradient to learn from: most rollouts fail early and teach little. This is the long-standing sparse-reward problem of RL, now with trajectories thousands of tokens long.
• Environment design. You can’t do agentic RL without an environment to act in (a sandboxed shell, a search index, a code runner), and it must be fast (you’ll run millions of rollouts), reproducible, and safe to let a half-trained model loose in. The environment is now part of the training stack, not an afterthought.
• Verifiable rewards for agents. Math and code gave clean verifiers. “Did the agent successfully book the trip / fix the bug / research the question” is far murkier: checking it is itself a hard problem, and a sloppy checker is an open invitation to reward hackingreward hackingWhen a policy finds ways to score high on the reward model without actually being better — exploiting quirks of an imperfect proxy. A central danger of RL post-training.See in glossary →.

Where the field is heading

The agentic-RL survey (Zhang et al., 2025) identifies several directions for this work: better automatic graders and AI-feedbackRLAIFReinforcement Learning from AI Feedback — replace human preference labels with labels from another model (or the model itself), making the feedback loop cheap and scalable.See in glossary → judges, reusable training environments built around browsers and coding tools, and methods for assigning credit across longer interactions. These tasks require evaluating whole trajectories, extending the response-level feedback common in RLHF and reasoning training.

Long interactions also create demands for better reward judges, efficient transfer from stronger models, and ways to manage the amount of context and reasoning used per task.