Driving · 2026-11-03

Gnomon4D: Learning Where and When to Focus for Geometric Motion-Grounded World-Action Modeling

Lingwei Dang, Yan Zhou, Juntong Li, Zhenhao Zhang, Yiteng Chen, Xiaoqiang Liu, Liang An, Hongwen Zhang, Wei Min, Guoxin Zhang, Pengfei Wan, Yebin Liu, Qingyao Wu

arXiv:2611.02474PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Gnomon4D is a world-action model that jointly generates future video, robot actions, and GeoDelta: a video-aligned 4D geometric motion field encoding image-plane displacement and depth change for embodiment and object points. It uses Interaction-Salient Attention (InSAttn) to choose where to read interactions and Transition-Anchored Aggregation (TransAgg) to choose when (around grasp/release), and links action-free pretraining on EgoManip-5000H to robot mid-training on Robot-30H.

Why it matters

The authors argue existing world-action models’ action attention underweights embodiment–object interactions and that RGB futures entangle state changes with appearance and lack 3D geometry. Gnomon4D aims to make interaction cues explicit to the action by reading a shared 4D motion field (GeoDelta) rather than relying on appearance. The reported results claim leads on RoboTwin 2.0, VLABench, and RoboDojo, including under visual shifts.

Method

  • Joint generation of future video V, robot action chunk A, and GeoDelta G (Δu, Δv, Δd) as a video-aligned 4D geometric motion field over embodiment and manipulated objects.
  • Interaction readout: InSAttn selects interaction regions by task-conditioned embodiment–object relative motion, and TransAgg selects transition frames by pooling around frame-to-frame changes (e.g., grasp/release).
  • Embodiment-agnostic pretraining: GeoDelta links action-free pretraining on curated EgoManip-5000H to robot mid-training on Robot-30H; GeoDelta is described as a human–robot quasi-action.
Abstract (from arXiv)

World-action models (WAMs) predict scene changes to guide robot actions, yet their RGB futures entangle state changes with appearance and lack 3D geometry. Recent WAMs add geometry as a training signal or a control interface, or build native 3D models, but decide what geometry to model rather than how the action should read it. Intuitively, control-relevant cues lie at embodiment--object interactions and their transitions, yet probing existing WAMs reveals that their action attention underweights these interactions in favor of the rest of the embodiment. To focus the action on these interactions, we present Gnomon4D, a WAM that jointly generates future video, robot actions, and GeoDelta, a video-aligned 4D geometric motion field recording the image-plane displacement and depth change of embodiment and object points. With the embodiment and objects in one field, their relative motion, which marks an interaction, can be read off. Guided by this motion and the task, Interaction-Salient Attention (InSAttn) selects where the action reads, and Transition-Anchored Aggregation (TransAgg) selects when, around grasp and release. As GeoDelta encodes surface motion rather than embodiment-specific controls, it serves as a human--robot quasi-action that links action-free pretraining on EgoManip-5000H, which we curate, to robot mid-training on Robot-30H. Gnomon4D leads on RoboTwin~2.0, VLABench, and RoboDojo, and outperforms real-robot baselines, especially under visual shifts. Project page: https://droliven.github.io/Gnomon4D.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-02 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

Gnomon4D: Learning Where and When to Focus for Geometric Motion-Grounded World-Action Modeling

Lingwei Dang, Yan Zhou, Juntong Li, Zhenhao Zhang, Yiteng Chen, Xiaoqiang Liu, Liang An, Hongwen Zhang, Wei Min, Guoxin Zhang, Pengfei Wan, Yebin Liu, Qingyao Wu

Preprint posted: 2026-11-02 · Original record ↗

arxiv
2611.02474
Explicit source versions (1)
  • v1 · 2026-11-02
  1. Preprint available

    Source date: 2026-11-02 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

World-action models (WAMs) predict scene changes to guide robot actions, yet their RGB futures entangle state changes with appearance and lack 3D geometry. Recent WAMs add geometry as a training signal or a control interface, or build native 3D models, but decide what geometry to model rather than how the action should read it. Intuitively, control-relevant cues lie at embodiment--object interactions and their transitions, yet probing existing WAMs reveals that their action attention underweights these interactions in favor of the rest of the embodiment. To focus the action on these interactions, we present Gnomon4D, a WAM that jointly generates future video, robot actions, and GeoDelta, a video-aligned 4D geometric motion field recording the image-plane displacement and depth change of embodiment and object points. With the embodiment and objects in one field, their relative motion, which marks an interaction, can be read off. Guided by this motion and the task, Interaction-Salient Attention (InSAttn) selects where the action reads, and Transition-Anchored Aggregation (TransAgg) selects when, around grasp and release. As GeoDelta encodes surface motion rather than embodiment-specific controls, it serves as a human--robot quasi-action that links action-free pretraining on EgoManip-5000H, which we curate, to robot mid-training on Robot-30H. Gnomon4D leads on RoboTwin~2.0, VLABench, and RoboDojo, and outperforms real-robot baselines, especially under visual shifts. Project page: https://droliven.github.io/Gnomon4D.