Latent Dynamics · 2026-10-28
RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
Haoyu Zhao, Yinzhou Tang, Zixuan Wang, Minghao Zhu, Dongchi Huang, Xingyue Zhao, Qin Zhao, Kehan Li, Siteng Huang, Xin Li, Zhenyu Zeng, Mingxiu Chen, Deli Zhao, Zhongyu Li
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
RynnWorld-Latent is a cross-embodiment, text-free video world model conditioned on 3D-grounded continuous latent actions from RynnLAM. It jointly pretrains on 23 datasets (~553.5M frames) and is post-trained to target robots. The model supports (1) digital teleoperation to generate action-video data and (2) latent-action synthesis via first-frame editing for cross-embodiment rollouts, improving prediction and policy success in evaluated settings.
Why it matters
The authors address an interface mismatch: robot embodiments have incompatible action spaces, while human videos lack native control labels. By using a shared latent-action representation as conditioning, RynnWorld-Latent avoids aligning raw robot action spaces while enabling controllable video prediction and generation for downstream robot learning.
Method
- Uses frozen RynnLAM to extract 3D-grounded continuous latent actions (motion, global action latent, camera component) as a shared conditioning interface for heterogeneous videos.
- Builds on a pretrained video transformer and conditions future video latents with global action tokens plus an identity-initialized, frame-aligned local pathway (action descriptors align to future latent-frame transitions).
- Provides two generation modes: digital teleoperation (target robot commands → action-video demonstrations) and latent-action synthesis (edit first frame → generate rollout conditioned on extracted latent actions).
Abstract (from arXiv)
Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.
Related papers
- OneLA: Learning Many Worlds from Latent Actions
- UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation
- Cross-Embodiment Robot Foundation World Models with Latent Actions
- Being-M0.7: A Latent World-Action Model for Humanoid Robots
- TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC
RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
Preprint posted: 2026-10-27 · Original record ↗
- arxiv
- 2610.36791
Explicit source versions (1)
- v1 · 2026-10-27
- Preprint available
Source date: 2026-10-27 · First detected: 2026-10-28 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.36791v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.