Robotics · 2026-10-14
Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates
Ryan Gao, Ashvin Anilkumar, Nicholas Corrado, William Cong, Josiah Hanna
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study real-robot RL without high-quality state estimates for a NAO soccer dribbling task under partial observability. They run an unscented Kalman filter (UKF) onboard and add RSSM-RLPD: a recurrent state-space model whose latent belief is corrected only when the ball is observed. On hardware, the robot learns to score from multiple starts after 30 minutes, reaching 100% success vs 0% for feed-forward RLPD under the same budget.
Why it matters
They address a practical obstacle for real-world RL: the learner must rely on noisy state estimates that can corrupt both policy inputs and rewards. RSSM-RLPD integrates an observation-gated recurrent world-model-style latent belief with an existing on-robot learning method (RLPD), aiming to make RL robust when key state (ball location) is intermittently unobserved.
Method
- Estimate key state on-board with a UKF and a Kalman-filter ball tracker, with no external instrumentation.
- Use RSSM-RLPD: an RSSM latent belief updated via a posterior correction only when the ball is observed; otherwise it advances via the prior.
- Train the RSSM with predictive losses from replay plus critic-guided representation learning, alongside RLPD’s replay-mixture approach.
Limitation
The authors note that they used a fixed demonstration dataset collected on the robot, so learning may depend on its quality and coverage; future work could measure sensitivity to demonstration quality and quantity and incorporate targeted human corrections.
Abstract (from arXiv)
A longstanding goal for robot reinforcement learning (RL) is that robots learn from their own physical experience, as humans and animals do. A key obstacle to learning directly from physical experience, especially outside instrumented lab environments, is that the state of the world is usually unobservable to the learner and must be estimated from on-board sensing. Errors in that estimate corrupt both the policy input and potentially the reward signal. We study this challenge using the contact-rich task of a bipedal robot learning to score soccer goals. We estimate all key state variables on-board with an unscented Kalman filter (UKF) and no external instrumentation (e.g., motion capture or overhead cameras). Because the UKF may incorrectly estimate state when the ball is unobserved, we learn a new state representation with a recurrent state-space model (RSSM) that, critically, only corrects the RSSM's latent state when the ball is observed. Building on RLPD, we introduce RSSM-RLPD and apply it to the task of a NAO robot learning to dribble a soccer ball into a goal. We show that the robot learns to score from multiple starting positions after 30 minutes of real-world interaction, reaching 100% success against 0% for a feed-forward policy under the same training budget. We complement our physical robot experiments with a simulation study and show that RSSM-RLPD learns policies that achieve a higher success rate than behavior cloning, a feed-forward, and GRU baseline. Taken together, our results show that the addition of a learned state representation to existing on-robot learning methods enables RL from real-robot data even with a high degree of partial observability.
Related papers
- DayDreamer: World Models for Physical Robot Learning
- EmbodiedRSI: Active Continual Robot Learning Through Hypothesis-Guided Co-Evolution
- Towards Agentic Robots: Model, Data, Environment, and Harness
- Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning
- Learning Robot Behavior Priors from Non-Robotic Synthetic Data
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-14 UTC
Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.14042
Explicit source versions (1)
- v1 · 2026-10-02
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
A longstanding goal for robot reinforcement learning (RL) is that robots learn from their own physical experience, as humans and animals do. A key obstacle to learning directly from physical experience, especially outside instrumented lab environments, is that the state of the world is usually unobservable to the learner and must be estimated from on-board sensing. Errors in that estimate corrupt both the policy input and potentially the reward signal. We study this challenge using the contact-rich task of a bipedal robot learning to score soccer goals. We estimate all key state variables on-board with an unscented Kalman filter (UKF) and no external instrumentation (e.g., motion capture or overhead cameras). Because the UKF may incorrectly estimate state when the ball is unobserved, we learn a new state representation with a recurrent state-space model (RSSM) that, critically, only corrects the RSSM's latent state when the ball is observed. Building on RLPD, we introduce RSSM-RLPD and apply it to the task of a NAO robot learning to dribble a soccer ball into a goal. We show that the robot learns to score from multiple starting positions after 30 minutes of real-world interaction, reaching 100% success against 0% for a feed-forward policy under the same training budget. We complement our physical robot experiments with a simulation study and show that RSSM-RLPD learns policies that achieve a higher success rate than behavior cloning, a feed-forward, and GRU baseline. Taken together, our results show that the addition of a learned state representation to existing on-robot learning methods enables RL from real-robot data even with a high degree of partial observability.