Robotics · 2026-11-10
Action Representation for Spatial Generalization in Robot World Models
Yuhang Sun
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study how action coordinate representations affect spatial generalization in a controlled robot peeling world-model benchmark. Using matched action–state inputs across world-, object-, and surface-relative frames, they find costs from limited data when equivalent interactions use different coordinate descriptions, irreducible ambiguity when peel thickness is removed, and improved consequence prediction from surface-relative coordinates under accurate local observations—yet vulnerability remains to coordinate-frame uncertainty.
Why it matters
Action representation determines whether learned action–consequence relationships transfer across spatial configurations. The paper separates consequence-preserving invariance from predictive sufficiency and shows that losing necessary physical information (e.g., peel thickness) creates input ambiguity that training cannot fix. It also distinguishes frame-estimation noise from incorrect contact geometry and execution deviations, linking representation choices to both prediction and decision-quality effects.
Method
- Use a deterministic geometric peeling simulator with a short-horizon blade stroke and fruit-attached material-removal outcomes, predicting consequence Y from an encoded state–action input via Ŷ = Wθ(R(X,A)).
- Compare matched action representations in world, object-relative, and surface-relative coordinate frames, varying only coordinate descriptions while keeping surface geometry, normals, and peel thickness matched.
- Test consequence-preserving global transformations, contact relocation, peel-thickness withholding, and further perturbations separating estimated-frame changes from incorrect contact geometry and executed-tool deviations; include counterfactual action ranking via regret experiments.
Limitation
The analysis addresses prescribed short-horizon strokes and geometric removal outcomes rather than the full fracture, adhesion, sensing, or closed-loop force-control mechanics of robotic peeling.
Abstract (from arXiv)
Spatial generalization in robot world models depends on whether action representations capture the physical relationships that remain valid across changing spatial configurations. Yet different coordinates can describe the same interaction, whereas similar motions can produce different outcomes when local physical conditions change. How these distinctions affect learned prediction remains insufficiently understood. This study evaluates matched action representations defined in world, object, and surface coordinate frames using a controlled robotic peeling benchmark. Global pose and contact relocation experiments reveal a cost associated with limited training data when equivalent interactions receive different spatial descriptions. Removing information about peel thickness creates an irreducible ambiguity because identical predictive inputs can correspond to different material removal outcomes. Coordinates defined relative to the surface improve consequence prediction and counterfactual action selection under accurate local observations, but learned predictors remain vulnerable to uncertainty in the coordinate frame. Joint centering of the surface and tool removes sensitivity to translation of their common origin, whereas incorrect contact geometry and actual execution deviations introduce different failure mechanisms. These results show that effective spatial generalization depends on preserving the relationships between actions and states that determine physical consequences, rather than simply minimizing coordinate variation. This perspective provides a principled basis for designing robot world models that reuse physical knowledge across spatial configurations while retaining the distinctions essential to prediction and planning.
Related papers
- From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- UNITAS: A 3D-Native World Action Model for Embodied Manipulation
- GPWM: Modeling the World as 3D Particle Motion
- What Limits Cross-Task Generalization in World Action Models?
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-09 · Indexed by Pub-AI: 2026-11-10 UTC
Action Representation for Spatial Generalization in Robot World Models
Preprint posted: 2026-11-09 · Original record ↗
- arxiv
- 2611.11869
Explicit source versions (1)
- v1 · 2026-11-09
- Preprint available
Source date: 2026-11-09 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11869v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Spatial generalization in robot world models depends on whether action representations capture the physical relationships that remain valid across changing spatial configurations. Yet different coordinates can describe the same interaction, whereas similar motions can produce different outcomes when local physical conditions change. How these distinctions affect learned prediction remains insufficiently understood. This study evaluates matched action representations defined in world, object, and surface coordinate frames using a controlled robotic peeling benchmark. Global pose and contact relocation experiments reveal a cost associated with limited training data when equivalent interactions receive different spatial descriptions. Removing information about peel thickness creates an irreducible ambiguity because identical predictive inputs can correspond to different material removal outcomes. Coordinates defined relative to the surface improve consequence prediction and counterfactual action selection under accurate local observations, but learned predictors remain vulnerable to uncertainty in the coordinate frame. Joint centering of the surface and tool removes sensitivity to translation of their common origin, whereas incorrect contact geometry and actual execution deviations introduce different failure mechanisms. These results show that effective spatial generalization depends on preserving the relationships between actions and states that determine physical consequences, rather than simply minimizing coordinate variation. This perspective provides a principled basis for designing robot world models that reuse physical knowledge across spatial configurations while retaining the distinctions essential to prediction and planning.
