Video World Models · 2026-11-10
ForeAct: Geometry-Guided Action Learning from World Representations
Wei Lu, Jiaxi Zhang, Yunheng Wang, Taowen Wang, Yuetong Fang, Zhenhui Zhang, Shuning Zhang, Zizhao Yuan, Yixiao Feng, Zecui Zeng, Renjing Xu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
ForeAct repurposes a pretrained video world model as a single-pass, low-latency backbone for robotic manipulation. It processes the clean observation latent in one forward pass (no future-frame synthesis or iterative denoising) and introduces learnable geometry tokens supervised by next-frame relative depth via bidirectional self-attention. A lightweight geometry readout is used only during training and removed at deployment.
Why it matters
The authors address two gaps in prior video world action models: expensive iterative inference and lack of explicit spatial structure, and in geometry-centric policies: limited action-conditioned spatiotemporal dynamics. ForeAct provides future-aware, geometry-guided action representations while avoiding iterative diffusion/rectified-flow sampling and without requiring depth input or an external geometry backbone at deployment.
Method
- Single-step representation: encode the current RGB observation into a clean visual latent and run one pretrained video world-model forward pass (t_diff=0) to produce observation-anchored visual tokens for action decoding, avoiding future-frame synthesis/iterative denoising.
- Future geometry learning: insert learnable future-geometry tokens into the shared video token stream; use unmasked bidirectional self-attention so supervised geometry tokens shape shared visual features.
- Training-only supervision: predict next-frame relative depth using a geometry head with learnable tokens supervised in latent/pixel/spatial-gradient spaces; remove the geometry decoder/readout at inference, enabling geometry-aware control without depth input.
Abstract (from arXiv)
Robotic manipulation requires representations that capture both scene geometry and its evolution during interaction. Existing approaches have largely treated these two aspects separately. World action models capture such dynamics by coupling action prediction with future video generation, but their reliance on iterative diffusion or rectified-flow inference leads to high computational cost, while their representations lack explicit spatial structure. Explicitly modeling scene geometry typically requires auxiliary modules that increase both parameter count and computational cost. Conversely, geometry-centric policies primarily recover observed spatial structure and often lack explicit supervision for anticipating future geometric changes during interaction. To address these limitations, we introduce ForeAct, which repurposes a pretrained video world model as a single-pass representation backbone for low-latency control. ForeAct directly processes the clean latent of the current observation in one forward pass, eliminating future-frame synthesis and iterative denoising while retaining the backbone's pretrained spatiotemporal priors. Rather than introducing a separate geometry backbone, ForeAct uses learnable tokens supervised by next-frame relative depth to shape the shared video representation through bidirectional self-attention. This compact interface turns the spatial knowledge already encoded in the backbone into a future-aware representation for action generation, with little parameter overhead. The lightweight geometry readout is used only during training and removed at deployment, enabling low-latency, geometry-aware control without depth input or an external geometry backbone. Without additional large-scale pretraining, ForeAct achieves 96.6% average success on LIBERO, 63.5% on SimplerEnv-Bridge, and a 74.4% success rate across 90 real-world rollouts.
Related papers
- MVG-WAM: Multiple View Geometry-Aware World-Action Modeling for Robotic Manipulation
- E0.5: From Reconstructive to Future-Predictive 3D Foundations for Robotic World-Action Modeling
- Relational Geometry Forcing for World Action Models
- Gnomon4D: Learning Where and When to Focus for Geometric Motion-Grounded World-Action Modeling
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC
ForeAct: Geometry-Guided Action Learning from World Representations
First announced by arXiv: 2026-11-10 · Original record ↗
- arxiv
- 2611.11500
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11500v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Robotic manipulation requires representations that capture both scene geometry and its evolution during interaction. Existing approaches have largely treated these two aspects separately. World action models capture such dynamics by coupling action prediction with future video generation, but their reliance on iterative diffusion or rectified-flow inference leads to high computational cost, while their representations lack explicit spatial structure. Explicitly modeling scene geometry typically requires auxiliary modules that increase both parameter count and computational cost. Conversely, geometry-centric policies primarily recover observed spatial structure and often lack explicit supervision for anticipating future geometric changes during interaction. To address these limitations, we introduce ForeAct, which repurposes a pretrained video world model as a single-pass representation backbone for low-latency control. ForeAct directly processes the clean latent of the current observation in one forward pass, eliminating future-frame synthesis and iterative denoising while retaining the backbone's pretrained spatiotemporal priors. Rather than introducing a separate geometry backbone, ForeAct uses learnable tokens supervised by next-frame relative depth to shape the shared video representation through bidirectional self-attention. This compact interface turns the spatial knowledge already encoded in the backbone into a future-aware representation for action generation, with little parameter overhead. The lightweight geometry readout is used only during training and removed at deployment, enabling low-latency, geometry-aware control without depth input or an external geometry backbone. Without additional large-scale pretraining, ForeAct achieves 96.6% average success on LIBERO, 63.5% on SimplerEnv-Bridge, and a 74.4% success rate across 90 real-world rollouts.
