Latent Dynamics · 2026-11-04
DreamX: Video Diffusion Model is a Universal Latent Action Learner
Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong, Soowon Son, Fu-En Yang, Seungryong Kim
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DreamX repurposes a pretrained video diffusion transformer as a universal latent action learner. It appends learnable query tokens to extract compact latent-action tokens (compressing 1,200 frame tokens to 4 tokens per latent frame via a 300× reduction). Trained with future-frame reconstruction, it enables any-to-any video transfer and zero-shot human-to-robot action transfer, and improves compact latent-action policy learning. On LARYBench, it reports state-of-the-art among latent action models.
Why it matters
The authors aim to learn latent actions that are consistent across different executions of the same action while being invariant to embodiment and appearance, improving transfer between humans and robots. DreamX uses the spatiotemporal and generative priors of a pretrained video diffusion model so latent actions capture motion-relevant dynamics disentangled from visual appearance, and can be used both for motion transfer and as compact prediction targets for downstream policy learning.
Method
- Append learnable query tokens to a pretrained video diffusion backbone’s spatiotemporal tokens to extract latent-action tokens for each latent frame.
- Supervise latent actions by decoding future frames from latent actions plus the initial frame (optionally with an auxiliary robot action task when labels exist), using the diffusion model’s generative prior.
- For control, compress latent-action tokens into low-dimensional codes via a variational bottleneck and train an observation- and instruction-conditioned robot policy to predict these codes plus low-level actions.
Abstract (from arXiv)
Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.
Related papers
- RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
- SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models
- SUAVE: Unified Video-Action Models via Masked Diffusion
- OneLA: Learning Many Worlds from Latent Actions
- UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-03 · Indexed by Pub-AI: 2026-11-04 UTC
DreamX: Video Diffusion Model is a Universal Latent Action Learner
Preprint posted: 2026-11-03 · Original record ↗
- arxiv
- 2611.05377
Explicit source versions (1)
- v1 · 2026-11-03
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-04 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.05377v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.
