Video World Models · 2026-11-11
OnlyAct: Extracting Action Representations from Foundation Models for World Models
Runjia Li, Minghao Chen, Junyu Xie, Philip Torr, Andrea Vedaldi, Tomas Jakab
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
OnlyAct extracts action representations from a pretrained video foundation model (V-JEPA 2) using label-free contrastive adaptation. It constructs positives by keeping motion while swapping scenes (flow-guided cut-and-paste) and uses hard negatives from the same video with different motion. The adapted encoder conditions an action-conditioned autoregressive video world model, enabling action transfer and real-action control on robot, camera-navigation, game, and driving benchmarks.
Why it matters
The authors target a key bottleneck in video world models: scalable action supervision is limited. Instead of learning latent actions from scratch with a reconstruction bottleneck, OnlyAct adapts pretrained spatiotemporal embeddings to isolate action while keeping scene invariance and expressiveness, and evaluates representations and resulting world-model control across multiple domains, including unseen datasets.
Method
- Start from a pretrained V-JEPA 2 encoder and encode consecutive frame pairs as a single action embedding (no reconstruction bottleneck).
- Label-free contrastive adaptation: positives keep motion but swap scene via optical-flow cut-and-paste (or alternative photometric augmentation for moving cameras); hard negatives come from other frame pairs in the same video with different motion.
- Freeze the adapted encoder and train an autoregressive action-conditioned video world model to predict the next frame from past frames and the extracted action stream; evaluate action expressiveness, scene invariance, and action-conditioned generation quality.
Limitation
The paper text provided does not state a specific limitation section with an explicit claim.
Abstract (from arXiv)
Video world models promise general-purpose interactive simulators, but their training is limited by action supervision: action labels are scarce, fragmented across incompatible specifications, and do not scale with the unlabeled video pretraining corpus. A common workaround jointly trains an action encoder to compress consecutive frames into a latent action and a decoder to reconstruct the next frame from the previous frame and that latent, forcing the latent to capture only new information. This has two limitations: learning the action space from scratch can limit generalization to unseen domains, and the bottleneck trades expressiveness for separation, with tighter settings losing nuance and looser ones risking scene leakage under domain shifts. We argue for a different starting point: rather than learning actions from scratch, we start from a pretrained foundation model (V-JEPA 2), whose embeddings of a frame pair already carry substantial action information after large-scale, diverse video pretraining. The remaining challenge is to extract the action while discarding the scene. We propose a label-free contrastive adaptation framework that pulls together frame pairs sharing the same motion under a different scene and pushes apart pairs from the same scene with different motion, reshaping the geometry of the pretrained space without imposing a reconstruction bottleneck. We introduce a unified evaluation protocol along three axes (action expressiveness, scene invariance, and action-conditioned video generation quality) across robot manipulation, camera navigation, game, and driving benchmarks, most of which are unseen during adaptation. Compared with annotation-free latent-action methods, our representations are substantially more expressive while matching their scene invariance, and they yield world models that transfer demonstrated actions to new scenes more faithfully.
Related papers
- Native Action-Prior Learning from Videos for World Action Models
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
- Cycle WAM: Coupling Action and Video through Cycle Consistency
- WorldBridge: Bridging Video and Action for Task Generalization in World Action Models
- PrismWAM: Decoupling State Representations in World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-10 · Indexed by Pub-AI: 2026-11-11 UTC
OnlyAct: Extracting Action Representations from Foundation Models for World Models
Preprint posted: 2026-11-10 · Original record ↗
- arxiv
- 2611.14420
Explicit source versions (1)
- v1 · 2026-11-10
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-11 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.14420v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Video world models promise general-purpose interactive simulators, but their training is limited by action supervision: action labels are scarce, fragmented across incompatible specifications, and do not scale with the unlabeled video pretraining corpus. A common workaround jointly trains an action encoder to compress consecutive frames into a latent action and a decoder to reconstruct the next frame from the previous frame and that latent, forcing the latent to capture only new information. This has two limitations: learning the action space from scratch can limit generalization to unseen domains, and the bottleneck trades expressiveness for separation, with tighter settings losing nuance and looser ones risking scene leakage under domain shifts. We argue for a different starting point: rather than learning actions from scratch, we start from a pretrained foundation model (V-JEPA 2), whose embeddings of a frame pair already carry substantial action information after large-scale, diverse video pretraining. The remaining challenge is to extract the action while discarding the scene. We propose a label-free contrastive adaptation framework that pulls together frame pairs sharing the same motion under a different scene and pushes apart pairs from the same scene with different motion, reshaping the geometry of the pretrained space without imposing a reconstruction bottleneck. We introduce a unified evaluation protocol along three axes (action expressiveness, scene invariance, and action-conditioned video generation quality) across robot manipulation, camera navigation, game, and driving benchmarks, most of which are unseen during adaptation. Compared with annotation-free latent-action methods, our representations are substantially more expressive while matching their scene invariance, and they yield world models that transfer demonstrated actions to new scenes more faithfully.
