Latent Dynamics · 2026-10-15
OneLA: Learning Many Worlds from Latent Actions
Ziyu Zhu, Zhaoxu Sun, Fan Jiang, Chiyu Wang, Junfeng Ni, Mu Xu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
OneLA learns a shared continuous latent action space from unlabeled videos using a latent action model with a shared orthogonal motion dictionary and a common predictive objective (no cross-domain alignment losses during latent action pretraining). Lightweight adapters map native controls into this space; a diffusion world model predicts context-dependent transitions for controllable rollouts. The authors report improved action readout across domains and lower action cycle-transfer rotation error.
Why it matters
The authors target world-model controllability across heterogeneous domains where action annotations are scarce and control interfaces differ. By learning reusable latent actions directly from videos and providing a shared conditioning interface, OneLA aims to connect unlabeled video prediction to downstream command control without task-specific design or alignment losses during pretraining.
Method
- Pretrain a latent action model (inverse dynamics) on frame pairs without action labels, using a shared orthonormal motion dictionary plus reconstruction, KL, and sparsity regularization.
- Freeze the LAM and train lightweight MLPs that map native control commands into the learned latent-action space.
- Train a diffusion transformer conditioned on the initial frame and latent actions (LAM pseudo-actions for unlabeled clips; MLP-mapped controls for labeled clips), jointly predicting video and realized-transition latent actions using frozen-LAM codes as targets.
Abstract (from arXiv)
Learning world models across diverse domains calls for a unified control space, yet ac- tion interfaces vary across domains and action annotations are often scarce. Latent ac- tions learned directly from videos offer a promising way to establish such a space without explicit action supervision. However, learning shared latent actions across domains re- mains challenging, as variations in appearance, embodiment, and dynamics can entangle domain-specific visual cues with transferable transition structure. To this end, we intro- duce OneLA, a simple framework that learns continuous latent actions through a shared orthogonal dictionary and a common predictive objective, without task-specific designs or cross-domain alignment losses during latent action pretraining. These latent actions pro- vide a shared conditioning interface for world models, with lightweight adapters mapping native control commands into the same space. Conditioned on latent actions and the cur- rent visual state, the world model predicts context-dependent transitions for controllable rollout. OneLA improves action readout across domains and reduces action cycle-transfer rotation error by 24.8% over the strongest evaluated baseline. A joint-training comparison further shows a 20.5% improvement in Human Preference Score v3 (HPSv3) for real- scene generation. These results suggest a path toward general-purpose world models that learn from diverse videos and support flexible control through latent action space.
Related papers
- UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation
- Native Action-Prior Learning from Videos for World Action Models
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
- TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control
- ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-15 UTC
OneLA: Learning Many Worlds from Latent Actions
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.20160
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Learning world models across diverse domains calls for a unified control space, yet ac- tion interfaces vary across domains and action annotations are often scarce. Latent ac- tions learned directly from videos offer a promising way to establish such a space without explicit action supervision. However, learning shared latent actions across domains re- mains challenging, as variations in appearance, embodiment, and dynamics can entangle domain-specific visual cues with transferable transition structure. To this end, we intro- duce OneLA, a simple framework that learns continuous latent actions through a shared orthogonal dictionary and a common predictive objective, without task-specific designs or cross-domain alignment losses during latent action pretraining. These latent actions pro- vide a shared conditioning interface for world models, with lightweight adapters mapping native control commands into the same space. Conditioned on latent actions and the cur- rent visual state, the world model predicts context-dependent transitions for controllable rollout. OneLA improves action readout across domains and reduces action cycle-transfer rotation error by 24.8% over the strongest evaluated baseline. A joint-training comparison further shows a 20.5% improvement in Human Preference Score v3 (HPSv3) for real- scene generation. These results suggest a path toward general-purpose world models that learn from diverse videos and support flexible control through latent action space.