Robotics · 2026-10-09
Diagnosing and Recovering from Observation-Space Shift at Long-Horizon Skill Seams
Pranav Wagh, Yu Fang, Yue Yang, Mingyu Ding
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study Observation-Space Shift (OSS) in long-horizon robotic manipulation where chaining independently trained skills causes downstream failures at skill seams. Using privileged simulator resets, they attribute the dominant shift to displaced articulated scene state and secondary objects. They build a detect–restore–resume system; on BOSS-44 it raises full-chain success from 7.6% to 26.5%, while tested alternatives (resampling, Diffusion Policy, and world-model baselines) fail from the evaluated seam states.
Why it matters
Skill chaining can fail even when each skill is reliable alone. The paper provides a causal diagnosis of what shifts at seams and a test-time recovery strategy that restores the displaced scene before resuming the downstream policy. This reframes some composition failures as reversible scene side effects rather than issues that can be fixed by retrying, stronger policies, or world-model planning.
Method
- Privileged simulator interventions diagnose OSS: selectively reset robot arm state, downstream target object, and other scene components to measure recovery gains.
- Detect–restore–resume system: a task-progress monitor detects stall; a learned restoration policy returns displaced scene components to downstream support; seam-robust fine-tuning improves resume robustness.
- Learned restoration teacher is state-based (drawer articulation and secondary objects) and distilled to a camera-only student using RGB cameras plus 16-D proprioception.
Limitation
On hardware, the progress monitor is limited by exterior-camera observability: failed grasps or dropped objects are often invisible without wrist and gripper sensing. The paper also states restoration is defined only for drawer seams, so chains whose second seam displaces a free object remain unrecovered.
Abstract (from arXiv)
Long-horizon robotic manipulation is often built by chaining independently trained skills. Although each skill can be reliable in isolation, performance degrades sharply when skills are chained: each downstream skill must start from the state its predecessor leaves behind rather than from its training distribution. We study this failure mode, Observation-Space Shift (OSS), and ask what causes these skill-seam failures. Using privileged simulator resets, we find that the dominant shift comes from displaced scene state (e.g., an open drawer or secondary objects left behind by earlier skills), not from the robot's joint configuration or the object the downstream skill manipulates. To test this diagnosis, we build a fully learned detect-restore-resume system: a task-progress monitor detects the stall, a learned policy restores the displaced scene components, and seam-robust fine-tuning lets the skill resume. It recovers the seam where every tested alternative fails, which we treat as evidence for the diagnosis rather than as a general-purpose method. On the BOSS-44 benchmark, the system improves full-chain success from 7.6% to 26.5%, a 3.5x improvement over the base policy and 51% of a privileged restoration oracle, whereas best-of-K resampling, a Diffusion Policy, and world-model baselines fail to recover from the evaluated seam states. On a real Franka arm running a fine-tuned $\pi_{0.5}$ policy, the same monitor is limited by exterior-camera observability, yet closing the loop still recovers some otherwise-terminal failures, motivating wrist and gripper sensing. These results suggest that some long-horizon composition failures are better addressed by restoring the scene before resuming the policy than by retrying from an off-support state.
Related papers
- MOST: Memory of Object States and Task Progress for Long-Horizon Mobile Manipulation
- Detecting Silent Action-Interface Mismatch with Calibrated Relational World Models
- Acting Through Registers
- Utility-WAM: Correcting Future-Induced Action Changes
- RESETTLE: Robotic Recovery through Disagreement-Triggered Retrieval and Efficient Corrective Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-09 UTC
Diagnosing and Recovering from Observation-Space Shift at Long-Horizon Skill Seams
Preprint posted: 2026-10-07 · Original record ↗
- arxiv
- 2610.10810
Explicit source versions (1)
- v1 · 2026-10-07
- Preprint available
Source date: 2026-10-07 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Long-horizon robotic manipulation is often built by chaining independently trained skills. Although each skill can be reliable in isolation, performance degrades sharply when skills are chained: each downstream skill must start from the state its predecessor leaves behind rather than from its training distribution. We study this failure mode, Observation-Space Shift (OSS), and ask what causes these skill-seam failures. Using privileged simulator resets, we find that the dominant shift comes from displaced scene state (e.g., an open drawer or secondary objects left behind by earlier skills), not from the robot's joint configuration or the object the downstream skill manipulates. To test this diagnosis, we build a fully learned detect-restore-resume system: a task-progress monitor detects the stall, a learned policy restores the displaced scene components, and seam-robust fine-tuning lets the skill resume. It recovers the seam where every tested alternative fails, which we treat as evidence for the diagnosis rather than as a general-purpose method. On the BOSS-44 benchmark, the system improves full-chain success from 7.6% to 26.5%, a 3.5x improvement over the base policy and 51% of a privileged restoration oracle, whereas best-of-K resampling, a Diffusion Policy, and world-model baselines fail to recover from the evaluated seam states. On a real Franka arm running a fine-tuned $π_{0.5}$ policy, the same monitor is limited by exterior-camera observability, yet closing the loop still recovers some otherwise-terminal failures, motivating wrist and gripper sensing. These results suggest that some long-horizon composition failures are better addressed by restoring the scene before resuming the policy than by retrying from an off-support state.
