Latent Dynamics · 2026-11-04
FloMo: 3D Scene Flow as a World Action Model Intermediate for Learning from Human Video
Jeremy Collins, Namra Patel, Ayush Agarwal, Shitij Govil, Animesh Garg
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
FloMo argues that dense 3D motion mediates transfer from pretrained video generation to robot action prediction. It renders 3D scene flow as RGB and encodes it in the same latent space as a pretrained Wan 2.2 video model, fine-tuning the backbone to predict scene-flow latents and action tokens using flow-matching. Co-trained on action-free human video and action-labeled robot teleoperation, it improves in-distribution policy success and out-of-distribution generalization versus video prediction or joint video+motion prediction.
Why it matters
The authors show that changing only the prediction target (3D motion vs video pixels/latents) affects policy learning and generalization. They report that predicting motion alone improves success and out-of-distribution generalization, while adding a video prediction target lowers policy success. This supports the view that video pretraining transfers to control through motion rather than high-entropy appearance signals.
Method
- Render 3D scene flow (3D displacement in a canonical frame) as an RGB video; encode scene-flow renders with the frozen Wan 2.2 VAE so scene-flow tokens share the same latent format as video.
- Use a Wan 2.2 5B DiT backbone (LoRA fine-tuned; VAE and text encoder frozen) with a flow-matching objective to jointly denoise scene-flow latents and action tokens from an initial observation and language instruction.
- Train in two stages on a mixture of action-free egocentric human video (scene-flow supervision) and action-labeled bimanual robot teleoperation (scene-flow + action supervision), then fine-tune on robot data only.
Limitation
The paper text does not state an explicit limitation.
Abstract (from arXiv)
Pretrained video generation models distill internet-scale motion priors into robot policies, but full video is a redundant signal for action prediction. We argue that dense 3D motion is the medium through which video pretraining best transfers to robot actions, and that predicting motion rather than video affords better sample-efficiency for action prediction. FloMo is a 5B video-generation model fine-tuned to predict 3D scene-flow and action tokens. Rendering scene flow as RGB and encoding it into the same latent space as video enables a pretrained video generation backbone to efficiently adapt to predict 3D scene flow. Co-trained on action-free human video and action-labeled robot teleoperation data, our model benefits from human-video pretraining and generalizes to unseen objects, motion primitives, and tasks. Replacing motion prediction with a video prediction objective degrades out-of-distribution generalization, while adding a video prediction target alongside motion lowers policy success rather than improving it. When motion, video, and action are predicted jointly, action tokens attend far more to motion than to video, suggesting that predicting video wastes model capacity. Together, these results indicate that motion effectively mediates the transfer from video pretraining to action. More information can be found at flomo-wam.github.io.
Related papers
- WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
- Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models
- RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
- Native Action-Prior Learning from Videos for World Action Models
- UNITAS: A 3D-Native World Action Model for Embodied Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-03 · Indexed by Pub-AI: 2026-11-04 UTC
FloMo: 3D Scene Flow as a World Action Model Intermediate for Learning from Human Video
Preprint posted: 2026-11-03 · Original record ↗
- arxiv
- 2611.05062
Explicit source versions (1)
- v1 · 2026-11-03
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-04 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.05062v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Pretrained video generation models distill internet-scale motion priors into robot policies, but full video is a redundant signal for action prediction. We argue that dense 3D motion is the medium through which video pretraining best transfers to robot actions, and that predicting motion rather than video affords better sample-efficiency for action prediction. FloMo is a 5B video-generation model fine-tuned to predict 3D scene-flow and action tokens. Rendering scene flow as RGB and encoding it into the same latent space as video enables a pretrained video generation backbone to efficiently adapt to predict 3D scene flow. Co-trained on action-free human video and action-labeled robot teleoperation data, our model benefits from human-video pretraining and generalizes to unseen objects, motion primitives, and tasks. Replacing motion prediction with a video prediction objective degrades out-of-distribution generalization, while adding a video prediction target alongside motion lowers policy success rather than improving it. When motion, video, and action are predicted jointly, action tokens attend far more to motion than to video, suggesting that predicting video wastes model capacity. Together, these results indicate that motion effectively mediates the transfer from video pretraining to action. More information can be found at flomo-wam.github.io.
