Robotics · 2026-10-28
TAPESTRY: A Single Backbone for Video Generation and Robot Control
Jeremy A. Collins, Seungjae Lee, Rachanon Wachakorn, Krishnan Srinivasan, Animesh Garg, Vitor Campagnolo Guizilini, Paarth Shah
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TAPESTRY repurposes a pretrained text-conditioned video diffusion transformer as an action-generating policy by treating each robot action timestep as a single “patch” token in the video VAE latent space. Actions are projected with a lightweight linear layer + affine-free LayerNorm, then processed by the shared transformer. The model is trained with video, joint video+action, and action-only diffusion, and deployed with one DDIM step (no video generation).
Why it matters
The authors target faster robotic control by reusing pretrained video dynamics priors without generating video at inference or training a separate action head from scratch. The single shared transformer backbone and one-step action-token generation aim to reduce online inference cost while maintaining generalization across simulation benchmarks and real-world bimanual manipulation.
Method
- Action-as-video-patch tokenization: map each action timestep to one DiT token using a linear projection + affine-free LayerNorm, then reuse the backbone’s pretrained patchifying convolution and output heads.
- Three diffusion objectives with a shared backbone: video diffusion on robot/human videos; joint video+action diffusion on robot demonstrations with coupled timesteps and an asymmetric attention mask; action-only diffusion with video tokens dropped.
- One-step inference: at test time only the action pathway runs; generate a compact action token sequence in a single DDIM step from pure noise (t=T to t=0), avoiding video decoding.
Limitation
The authors observe that multi-step DDIM inference produces high-frequency jitter in action chunks, degrading robot execution, and report that single-step inference avoids this effect; multi-step jitter is present for repurposed video-generation backbones.
Abstract (from arXiv)
Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at https://tapestry-wam.github.io.
Related papers
- From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models
- SUAVE: Unified Video-Action Models via Masked Diffusion
- RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
- W2A: Distilling World Priors into Action Models for Real-Time Robotic Control
- Sparse-WAM: Accelerating World Action Models via Action-Guided Sparse Imagination
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC
TAPESTRY: A Single Backbone for Video Generation and Robot Control
Preprint posted: 2026-10-27 · Original record ↗
- arxiv
- 2610.36823
Explicit source versions (1)
- v1 · 2026-10-27
- Preprint available
Source date: 2026-10-27 · First detected: 2026-10-28 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.36823v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at https://tapestry-wam.github.io.
