Robotics · 2026-11-11

From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models

Guoqiang Liang, Yiqi Lin, Yanzhe Chen, Kevin Yuchen Ma, Mike Zheng Shou

arXiv:2611.14365PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

MetaWAM is a world action model that injects robot signals (proprioception, actions, values, task type) into a single unified “meta-frame” for a pretrained video diffusion model. Instead of separate latent frames or learned encoders/decoders, it packs signals into disjoint DCT frequency channel bands and lifts them via inverse DCT. On RoboCasa it reports 68.9% success and reports 39% lower inference latency; it reports 85.0% success on three real-world Franka tasks.

Why it matters

The authors argue robot-signal injection is a distribution-alignment problem with respect to the pretrained VDM denoiser’s prior. Their diagnostic uses relative reconstruction error of a frozen denoiser to show frequency-domain packing requires fewer corrections than spatial encodings. MetaWAM then uses a decoder-free analytic interface (DCT/iDCT readout) to reduce redundant latent frames while keeping the pretrained denoising architecture unchanged.

Method

  • Diagnose VDM-prior compatibility by measuring relative reconstruction error of a frozen Cosmos-Predict2 denoiser on injected latents across noise levels.
  • Propose MetaWAM: pack all robot signals into one unified meta-frame by assigning each signal to disjoint frequency-domain channel bands and lifting with orthonormal 2D inverse DCT.
  • Use slot-aware, band-level routing to train policy, world-model, and value prediction with one denoising model, recovering signals analytically via fixed slot readout (no learned decoding head).

Limitation

Robustness is uneven: the authors report lower performance on LIBERO-Plus background (72.68%) and robot initial-state (partially shown).

Abstract (from arXiv)

Pretrained video diffusion models (VDMs) offer a powerful foundation for World Action Models (WAMs), which jointly predict robot actions and future observations. Building such models requires injecting low-dimensional robot signals (e.g. proprioception, actions, and values) into a latent space shaped by natural videos, and a key design choice is the interface through which these signals enter: existing schemes treat it as a formatting detail, spending a separate latent frame on each signal or bolting on dedicated encoders and decoders. We argue that signal injection is instead a distribution-alignment problem: how well a representation matches the prior of the pretrained VDMs during denoising. Using the relative reconstruction error of a frozen VDM denoiser as a probe, we find that conventional spatial encodings sit far off this prior, whereas frequency-domain packing yields latents that the denoiser corrects far less. Motivated by this diagnostic result, we propose MetaWAM, which folds all robot signals into a single unified meta-frame: each signal is assigned to a disjoint frequency-domain channel band and lifted to the spatial latent by an inverse DCT, yielding a decodable interface aligned with the pretrained VDM prior without any added module or architectural change. A fixed band layout together with slot-aware routing lets the same meta-frame support policy, world-model, and value prediction over a shared set of robot variables. On RoboCasa, MetaWAM reaches 68.9% success while collapsing the per-signal frames into one and cutting inference latency by 39%, and attains 85.0% success on three real-world Franka tasks. Code and models are available at https://github.com/showlab/MetaWAM

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-10 · Indexed by Pub-AI: 2026-11-11 UTC

preprint

From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models

Guoqiang Liang, Yiqi Lin, Yanzhe Chen, Kevin Yuchen Ma, Mike Zheng Shou

Preprint posted: 2026-11-10 · Original record ↗

arxiv
2611.14365
Explicit source versions (1)
  • v1 · 2026-11-10
  1. Preprint available

    Source date: 2026-11-10 · First detected: 2026-11-11 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Pretrained video diffusion models (VDMs) offer a powerful foundation for World Action Models (WAMs), which jointly predict robot actions and future observations. Building such models requires injecting low-dimensional robot signals (e.g. proprioception, actions, and values) into a latent space shaped by natural videos, and a key design choice is the interface through which these signals enter: existing schemes treat it as a formatting detail, spending a separate latent frame on each signal or bolting on dedicated encoders and decoders. We argue that signal injection is instead a distribution-alignment problem: how well a representation matches the prior of the pretrained VDMs during denoising. Using the relative reconstruction error of a frozen VDM denoiser as a probe, we find that conventional spatial encodings sit far off this prior, whereas frequency-domain packing yields latents that the denoiser corrects far less. Motivated by this diagnostic result, we propose MetaWAM, which folds all robot signals into a single unified meta-frame: each signal is assigned to a disjoint frequency-domain channel band and lifted to the spatial latent by an inverse DCT, yielding a decodable interface aligned with the pretrained VDM prior without any added module or architectural change. A fixed band layout together with slot-aware routing lets the same meta-frame support policy, world-model, and value prediction over a shared set of robot variables. On RoboCasa, MetaWAM reaches 68.9% success while collapsing the per-signal frames into one and cutting inference latency by 39%, and attains 85.0% success on three real-world Franka tasks. Code and models are available at https://github.com/showlab/MetaWAM