Robotics · 2026-10-22
AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation
Likui Zhang, Zhe Zhu, Zhihao Zhan, Yibo Bai, Hanlin Wang, Dongwei Li, Bin Wang, Tianle Zhang, Xiaodan Liang, Liang Lin
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
AtomicRL is an RL post-training framework for long-horizon robotic manipulation with vision-language-action policies. It uses SAPO to gate advantage/TD-residual propagation at atomic subtask semantic boundaries to reduce credit contamination from failed later subtasks, and BATE to sample tasks and adjust exploration based on subtask mastery. It also adds an automated Real2Sim2Real pipeline for real-robot transfer.
Why it matters
The authors target two issues in VLA-RL for long-horizon manipulation: delayed/biased credit assignment from trajectory-level sparse rewards and wasted exploration on already-mastered subtasks. By structuring tasks into atomic subtasks and using semantic boundaries to regulate credit propagation, plus bottleneck-aware targeted exploration, AtomicRL improves success/sequence completion in both simulation and real-world robot tasks.
Method
- SAPO (Semantic-Aware Policy Optimization): uses semantic subtask boundaries from PCC to attenuate cross-subtask advantage propagation when downstream atomic subtasks fail, mitigating credit contamination while preserving within-subtask credit assignment.
- BATE (Bottleneck-Adaptive Target Exploration): estimates mastery via an exponential moving average of completion rates and uses it to adapt full-task sampling and subtask-level entropy regularization toward bottleneck subtasks.
- Real2Sim2Real pipeline: reconstructs an interactive simulation environment from a single real-world multimodal demonstration (plus required auxiliary info) for simulation RL and then deploys the trained policy on physical robots.
Limitation
AtomicRL does not fundamentally address the overall efficiency of online RL training; online RL still requires substantial environment interaction and training time, particularly in real-robot settings where interaction and exploration costs remain high.
Abstract (from arXiv)
Vision-language-action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance. To address these challenges, we propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration. AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping. In addition, Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives based on the mastery of individual atomic subtasks, directing learning toward the remaining bottlenecks. Finally, we develop a Real2Sim2Real pipeline that bridges online RL in simulation with real-world deployment, facilitating practical transfer to physical robots. On LIBERO-LONG, AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO across two VLA architectures. On CALVIN ABC-D, it improves full-task performance by up to 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by an average of 21.3%. These results demonstrate that AtomicRL effectively improves the learning efficiency and manipulation performance of VLA policies on long-horizon tasks.
Related papers
- DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
- MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
- CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
- Fast-VLA: You Only Think Once for Robot Manipulation
- EvoMem-VLA: State-Evolution Memory for Long-Horizon Robot Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-22 UTC
AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation
Preprint posted: 2026-10-21 · Original record ↗
- arxiv
- 2610.30097
Explicit source versions (1)
- v1 · 2026-10-21
- Preprint available
Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision-language-action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance. To address these challenges, we propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration. AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping. In addition, Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives based on the mastery of individual atomic subtasks, directing learning toward the remaining bottlenecks. Finally, we develop a Real2Sim2Real pipeline that bridges online RL in simulation with real-world deployment, facilitating practical transfer to physical robots. On LIBERO-LONG, AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO across two VLA architectures. On CALVIN ABC-D, it improves full-task performance by up to 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by an average of 21.3%. These results demonstrate that AtomicRL effectively improves the learning efficiency and manipulation performance of VLA policies on long-horizon tasks.