ARPO (Agentic Reinforced Policy Optimization): agentic reinforcement learning for a tool-using small model
I implemented ARPO (Agentic Reinforced Policy Optimization, arXiv 2507.19849) for Qwen3-0.6B in August 2025: supervised fine-tuning followed by reinforcement learning, extra rollouts branched at high-uncertainty steps, credit assigned to individual tool calls, and the Muon optimizer. The first combined SFT and RL run completed, and most of the later work went into the numerical stability that RL on tool use needs.
ARPO is a reinforcement-learning method for agents that use tools over many turns. Its idea is to spend exploration where the model is unsure: right after a tool result, when the next step could go several ways.
What I implemented
- Two stages: supervised fine-tuning to teach the tool format, then reinforcement learning on top.
- Entropy-based adaptive rollouts: when the model's uncertainty spikes after a tool call, the rollout branches there instead of only at the start.
- Step-level advantage attribution: credit goes to the specific tool-use actions that mattered, not just to the whole trajectory.
- The Muon optimizer, with orthogonalised updates, alongside the usual AdamW baseline.
- Hardware range: from a 16 GB consumer GPU with QLoRA up to rented H200 and B200 machines.
RL on the stack labs use
- ARPO on Prime Intellect H200s, run with verifier environments from their ecosystem: the same kind of environments labs buy.
- The POLARIS recipe on Prime-RL, reproduced for math reasoning on Qwen3-0.6B.
- AvataRL with SAC, an experiment in hybrid RL for training a language model, where a second agent makes temporary edits to the model's weights during training.
No results are claimed here for these runs.
What the work taught
- RL on tool use is numerically fragile. Most of the later commits were stability fixes, and they were documented as they landed so the next run didn't repeat them.
- Pin the boring dependencies. Several training failures came from library versions, not the method.
Facts
| Item | Detail |
|---|---|
| Method | ARPO, arXiv 2507.19849 |
| Model | Qwen3-0.6B |
| History | 7 commits, Aug 2025; first SFT + RL run completed on Aug 15 |
| Results | No benchmark numbers claimed here |
What you can buy
Agentic RL environments built to your spec: multi-turn tool-use tasks with verifiers, ready for SFT-then-RL pipelines like this one.
FAQ
What is ARPO?
Agentic Reinforced Policy Optimization: an RL method for language-model agents that branches extra rollouts at high-uncertainty steps after tool calls and assigns credit to individual tool-use actions.
Why start with supervised fine-tuning?
So the model already produces valid tool calls before RL begins. RL then improves which calls it makes, instead of spending its budget learning the format.
License it, or have me build an environment in the same style.