Driving · 2026-10-05

XGenAct: Geometry-Enhanced World Action Models through Cross-Task Generation

Tingting Du, Ziyao Wang, Guoheng Sun, Ang Li

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

XGenAct is a world action model that generates RGB, robot actions, metric depth, surface normals, and functional role segmentation as RGB video streams using deterministic codecs. Training samples pair one perception modality with action, while a single video diffusion transformer and objective handle all tasks without modality-specific learned heads. On held-out RLBench, it achieves 52% average closed-loop success vs 26% for the strongest evaluated baselines.

Why it matters

The authors target a bottleneck in world action models: predicting only 2D RGB futures without explicit spatial understanding needed for manipulation. XGenAct unifies multiple geometric/semantic supervision signals (depth, normals, functional segmentation) and action generation in one shared video-generation space, aiming to improve both closed-loop robot control and future geometry/semantics prediction accuracy.

Method

Represents RGB observations, 7-DoF robot actions, metric depth, surface normals, and functional role segmentation as RGB videos via deterministic codecs, then encodes all streams with the same frozen video VAE into a shared latent space.
During training, samples select one perception modality template paired with action (not all perception modalities at once); all templates share one diffusion transformer, output head, and flow matching objective, with no modality-specific learned heads or auxiliary losses.
Uses diffusion flow-matching over packed latent sequences with conditioning plans that specify which segment frames are observed vs generated; applies the reconstruction loss only to positions that must be generated.

Abstract (from arXiv)

World action models (WAMs) have advanced robot control by predicting how observations and actions evolve over time. Despite this progress, RGB and action based future prediction does not explicitly address the spatial understanding needed for robot manipulation. Existing efforts often add a limited set of spatial prediction tasks through specialized heads or branches, leaving both the range of spatial supervision and the model architecture fragmented. We introduce XGenAct, a world action model that represents RGB observations, robot actions, metric depth, surface normals, and functional role segmentation as RGB videos through deterministic codecs. By sampling perception and action tasks during training, XGenAct uses one video diffusion transformer and one objective to learn temporal prediction across these spaces without modality specific learned heads. On held out RLBench tasks, structured perception training improves average closed loop success over RGB only training, and XGenAct achieves 52% success in the five task external comparison, versus 26% for the strongest evaluated baselines. It also predicts future depth and segmentation more accurately than the evaluated pipelines that generate RGB first and then apply a frozen perception expert.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC

preprint