ARPO is a reinforcement-learning method for agents that use tools over many turns. Its idea is to spend exploration where the model is unsure: right after a tool result, when the next step could go several ways.
What I implemented
- Two stages: supervised fine-tuning to teach the tool format, then reinforcement learning on top.
- Entropy-based adaptive rollouts: when the model's uncertainty spikes after a tool call, the rollout branches there instead of only at the start.
- Step-level advantage attribution: credit goes to the specific tool-use actions that mattered, not just to the whole trajectory.
- The Muon optimizer, with orthogonalised updates, alongside the usual AdamW baseline.
- Hardware range: from a 16 GB consumer GPU with QLoRA up to rented H200 and B200 machines.
RL on the stack labs use
- ARPO on Prime Intellect H200s, run with verifier environments from their ecosystem: the same kind of environments labs buy.
- The POLARIS recipe on Prime-RL, reproduced for math reasoning on Qwen3-0.6B.
- AvataRL with SAC, an experiment in hybrid RL for training a language model, where a second agent makes temporary edits to the model's weights during training.
No results are claimed here for these runs.
What the work taught
- RL on tool use is numerically fragile. Most of the later commits were stability fixes, and they were documented as they landed so the next run didn't repeat them.
- Pin the boring dependencies. Several training failures came from library versions, not the method.
Facts
| Item | Detail |
|---|---|
| Method | ARPO, arXiv 2507.19849 |
| Model | Qwen3-0.6B |
| History | 7 commits, Aug 2025; first SFT + RL run completed on Aug 15 |
| Results | No benchmark numbers claimed here |
What you can buy
Agentic RL environments built to your spec: multi-turn tool-use tasks with verifiers, ready for SFT-then-RL pipelines like this one.
FAQ
What is ARPO?
Agentic Reinforced Policy Optimization: an RL method for language-model agents that branches extra rollouts at high-uncertainty steps after tool calls and assigns credit to individual tool-use actions.
Why start with supervised fine-tuning?
So the model already produces valid tool calls before RL begins. RL then improves which calls it makes, instead of spending its budget learning the format.