Video World Models · 2026-11-10

ForeAct: Geometry-Guided Action Learning from World Representations

Wei Lu, Jiaxi Zhang, Yunheng Wang, Taowen Wang, Yuetong Fang, Zhenhui Zhang, Shuning Zhang, Zizhao Yuan, Yixiao Feng, Zecui Zeng, Renjing Xu

arXiv:2611.11500PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

ForeAct repurposes a pretrained video world model as a single-pass, low-latency backbone for robotic manipulation. It processes the clean observation latent in one forward pass (no future-frame synthesis or iterative denoising) and introduces learnable geometry tokens supervised by next-frame relative depth via bidirectional self-attention. A lightweight geometry readout is used only during training and removed at deployment.

Why it matters

The authors address two gaps in prior video world action models: expensive iterative inference and lack of explicit spatial structure, and in geometry-centric policies: limited action-conditioned spatiotemporal dynamics. ForeAct provides future-aware, geometry-guided action representations while avoiding iterative diffusion/rectified-flow sampling and without requiring depth input or an external geometry backbone at deployment.

Method

  • Single-step representation: encode the current RGB observation into a clean visual latent and run one pretrained video world-model forward pass (t_diff=0) to produce observation-anchored visual tokens for action decoding, avoiding future-frame synthesis/iterative denoising.
  • Future geometry learning: insert learnable future-geometry tokens into the shared video token stream; use unmasked bidirectional self-attention so supervised geometry tokens shape shared visual features.
  • Training-only supervision: predict next-frame relative depth using a geometry head with learnable tokens supervised in latent/pixel/spatial-gradient spaces; remove the geometry decoder/readout at inference, enabling geometry-aware control without depth input.
Abstract (from arXiv)

Robotic manipulation requires representations that capture both scene geometry and its evolution during interaction. Existing approaches have largely treated these two aspects separately. World action models capture such dynamics by coupling action prediction with future video generation, but their reliance on iterative diffusion or rectified-flow inference leads to high computational cost, while their representations lack explicit spatial structure. Explicitly modeling scene geometry typically requires auxiliary modules that increase both parameter count and computational cost. Conversely, geometry-centric policies primarily recover observed spatial structure and often lack explicit supervision for anticipating future geometric changes during interaction. To address these limitations, we introduce ForeAct, which repurposes a pretrained video world model as a single-pass representation backbone for low-latency control. ForeAct directly processes the clean latent of the current observation in one forward pass, eliminating future-frame synthesis and iterative denoising while retaining the backbone's pretrained spatiotemporal priors. Rather than introducing a separate geometry backbone, ForeAct uses learnable tokens supervised by next-frame relative depth to shape the shared video representation through bidirectional self-attention. This compact interface turns the spatial knowledge already encoded in the backbone into a future-aware representation for action generation, with little parameter overhead. The lightweight geometry readout is used only during training and removed at deployment, enabling low-latency, geometry-aware control without depth input or an external geometry backbone. Without additional large-scale pretraining, ForeAct achieves 96.6% average success on LIBERO, 63.5% on SimplerEnv-Bridge, and a 74.4% success rate across 90 real-world rollouts.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC

preprint

ForeAct: Geometry-Guided Action Learning from World Representations

Wei Lu, Jiaxi Zhang, Yunheng Wang, Taowen Wang, Yuetong Fang, Zhenhui Zhang, Shuning Zhang, Zizhao Yuan, Yixiao Feng, Zecui Zeng, Renjing Xu

First announced by arXiv: 2026-11-10 · Original record ↗

arxiv
2611.11500
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11500v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Robotic manipulation requires representations that capture both scene geometry and its evolution during interaction. Existing approaches have largely treated these two aspects separately. World action models capture such dynamics by coupling action prediction with future video generation, but their reliance on iterative diffusion or rectified-flow inference leads to high computational cost, while their representations lack explicit spatial structure. Explicitly modeling scene geometry typically requires auxiliary modules that increase both parameter count and computational cost. Conversely, geometry-centric policies primarily recover observed spatial structure and often lack explicit supervision for anticipating future geometric changes during interaction. To address these limitations, we introduce ForeAct, which repurposes a pretrained video world model as a single-pass representation backbone for low-latency control. ForeAct directly processes the clean latent of the current observation in one forward pass, eliminating future-frame synthesis and iterative denoising while retaining the backbone's pretrained spatiotemporal priors. Rather than introducing a separate geometry backbone, ForeAct uses learnable tokens supervised by next-frame relative depth to shape the shared video representation through bidirectional self-attention. This compact interface turns the spatial knowledge already encoded in the backbone into a future-aware representation for action generation, with little parameter overhead. The lightweight geometry readout is used only during training and removed at deployment, enabling low-latency, geometry-aware control without depth input or an external geometry backbone. Without additional large-scale pretraining, ForeAct achieves 96.6% average success on LIBERO, 63.5% on SimplerEnv-Bridge, and a 74.4% success rate across 90 real-world rollouts.