Driving · 2026-11-02
WorldBridge: Bridging Video and Action for Task Generalization in World Action Models
Jun Cen, Yubo Guo, Hao Li, Liangkai Li, Yaoxu Lyu, Meng Wei, Zeqi Xiao, Sizhe Yang, Yuqiang Yang, Yifan Ye, Shanghang Zhang, Duo Zheng
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
WorldBridge is a World Action Model that bridges video prediction and action generation via separate video/action experts connected through structured attention and trained with joint flow matching. By varying out-of-distribution video and action supervision, the authors show that adding OOD videos (no OOD action labels) and then adding sparse OOD action demonstrations improves task-transfer success on RoboTwin 2.0.
Why it matters
The paper studies how video generalization and action grounding contribute to task generalization in embodied settings, and uses these findings to design WorldBridge. The authors report large gains in out-of-distribution success by expanding video coverage and then adding sparse action labels, and they evaluate WorldBridge across multiple manipulation benchmarks and real-world platforms.
Method
- WorldBridge uses a Mixture-of-Transformers with separate video and action experts connected via joint self-attention; stage-specific attention masks create an explicit video-to-action information path for control.
- Training uses a joint flow-matching objective: video future latents and action chunks are independently noised with sampled flow timesteps, then denoised via learned velocity fields.
- The model adds visual memory at short- and long-term scales plus action forcing (token-wise action noise levels during training) and uses a unified data pipeline standardizing robot trajectories and egocentric hand-derived supervision.
Abstract (from arXiv)
Achieving the broad task generalization of embodied models in physical environments remains a central challenge in robot learning. We study this challenge in World Action Models (WAMs) through two complementary capabilities: video generalization, which predicts the visual evolution of new tasks, and action generalization, which translates these predictions into executable behavior through a video-to-action bridge. By varying video and action supervision for tasks outside an initial training set, we examine their respective contributions to task transfer. On RoboTwin 2.0, using only in-distribution video-action supervision yields a 25.07% success rate on out-of-distribution tasks. Introducing videos of these tasks without their action labels raises success to 55.90%, while adding just one action-labeled demonstration per task further improves it to 70.70%. These findings suggest that broader video coverage enables task transfer by leveraging existing inverse-dynamics capabilities, while even sparse action supervision further strengthens those capabilities. Guided by this perspective, we introduce WorldBridge, a WAM with separate video and action experts connected through structured attention and trained with joint flow matching. WorldBridge incorporates visual memory at two temporal scales and action forcing, alongside a unified data processing pipeline that standardizes and filters heterogeneous robot trajectories and egocentric hand-derived supervision. Across multiple manipulation benchmarks, WorldBridge achieves state-of-the-art performance among the compared methods, including 96.23% on RoboTwin 2.0 full data setting and 80.9% on clean to random setting, 82.8% on RoboCasa GR1 TableTop, and 57.6% on DOMINO Level-1. Our study highlights the complementary roles of video prediction and action grounding in developing robots that generalize across tasks.
Related papers
- OpenWAM: An Open Framework for Composable World-Action Models
- Video Prediction Policy 2: Predict Better, Act Better
- WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
- What Limits Cross-Task Generalization in World Action Models?
- Native Action-Prior Learning from Videos for World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-02 · Indexed by Pub-AI: 2026-11-02 UTC
WorldBridge: Bridging Video and Action for Task Generalization in World Action Models
First announced by arXiv: 2026-11-02 · Original record ↗
- arxiv
- 2610.40748
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-02 · First detected: 2026-11-02 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40748v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Achieving the broad task generalization of embodied models in physical environments remains a central challenge in robot learning. We study this challenge in World Action Models (WAMs) through two complementary capabilities: video generalization, which predicts the visual evolution of new tasks, and action generalization, which translates these predictions into executable behavior through a video-to-action bridge. By varying video and action supervision for tasks outside an initial training set, we examine their respective contributions to task transfer. On RoboTwin 2.0, using only in-distribution video-action supervision yields a 25.07% success rate on out-of-distribution tasks. Introducing videos of these tasks without their action labels raises success to 55.90%, while adding just one action-labeled demonstration per task further improves it to 70.70%. These findings suggest that broader video coverage enables task transfer by leveraging existing inverse-dynamics capabilities, while even sparse action supervision further strengthens those capabilities. Guided by this perspective, we introduce WorldBridge, a WAM with separate video and action experts connected through structured attention and trained with joint flow matching. WorldBridge incorporates visual memory at two temporal scales and action forcing, alongside a unified data processing pipeline that standardizes and filters heterogeneous robot trajectories and egocentric hand-derived supervision. Across multiple manipulation benchmarks, WorldBridge achieves state-of-the-art performance among the compared methods, including 96.23% on RoboTwin 2.0 full data setting and 80.9% on clean to random setting, 82.8% on RoboCasa GR1 TableTop, and 57.6% on DOMINO Level-1. Our study highlights the complementary roles of video prediction and action grounding in developing robots that generalize across tasks.