Robotics · 2026-10-06
ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies
Zhe Tao, Feiran Wang, Gaowen Liu, Ramana Rao Kompella$, Yan Yan
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
ForeAct3D is an action-conditioned semantic 3D future world modeling framework integrated into VLA policies. It decodes depth, 3-class semantics, and camera pose from policy features at current and future endpoints, grounds future queries on the policy-generated action chunk, and enforces physical consistency via background staticity, instance rigidity, and wrist-camera pose anchored to end-effector kinematics. No future prediction is required at inference.
Why it matters
The authors target a mismatch in VLA policies where future forecasts are decoupled from the actions the policy will execute and lack physical constraints. By predicting semantically grounded 3D states with action conditioning and cross-time physical regularization, ForeAct3D aims to make learned representations reflect planned interaction outcomes, improving manipulation success and future object localization.
Method
- Learnable geometric queries decode depth, three-class semantic masks, and camera pose at current and future semantic 3D states from VLA backbone features.
- Future queries are conditioned on the policy-generated action chunk to ground the predicted transition in the planned interaction, without rollout-time future prediction at inference.
- A physical-consistency closure enforces background staticity, instance-level rigidity (via rigid transforms), and end-effector kinematics for wrist-camera pose.
Limitation
ForeAct3D assumes static backgrounds and rigid instances, limiting applicability to deformable objects, articulated instances, and moving backgrounds. Same-pixel instance membership approximates temporal correspondence without tracking material points and becomes less reliable under large motion. Training requires dense geometric supervision and instance annotations, making predictions sensitive to depth and mask errors.
Abstract (from arXiv)
Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3\% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7\% to 37.8\% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.
Related papers
- WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
- Forward Dynamics Models
- Magic-W0: A Structured World-Action Foundation Model for Physical Intelligence
- PLaW-VLA: Predictive Latent World Modeling for Vision-Language-Action Policies
- Vision-Language-Delta-Action: Learning Compact Deltas as Actionable Transition Interfaces for Robotic Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-03 · Indexed by Pub-AI: 2026-10-06 UTC
ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies
Preprint posted: 2026-10-03 · Original record ↗
- arxiv
- 2610.04607
Explicit source versions (1)
- v2 · 2026-10-09
- Explicit source version detected · v2
Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
- Preprint available
Source date: 2026-10-03 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.