Video World Models · 2026-10-08
Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling
Ryan Solgi, Parsa Madinei, Zheng Zhang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Tucker bottleneck attention (TuBA) replaces quadratic self-attention over multidimensional token grids with attention over a compact low-rank Tucker core. TuBA reads a hidden tensor into Tucker cores using learned mode-wise factors, applies multi-head self-attention on the cores, then writes the update back to the original space. An autoregressive variant uses block-causal masking on core slots. The authors evaluate on video prediction and global weather forecasting.
Why it matters
The paper targets a scalability limitation of self-attention for long multidimensional sequences by using a low-rank tensor structure as an inductive bias for global token mixing. The authors report accuracy–efficiency improvements versus standard self-attention and other efficient attention methods on video and weather forecasting, along with stated subquadratic compute from Tucker-core attention.
Method
- TuBA treats hidden representations as multidimensional tensors, reads them into compact Tucker cores via learned mode-wise factors, applies standard multi-head self-attention on the core slots, then writes the update back to the ambient tensor space.
- Autoregressive TuBA imposes causality at the sequence-of-tensors level using block-causal masking on concatenated core slots: within a tensor, core slots may interact; each tensor attends only to itself and earlier tensors.
- Computational trade-off: Tucker core attention cost O(R^2 d + R d^2) plus mode-wise read/write contractions, yielding stated subquadratic complexity when R=o(N), with stated reduced FLOPs and activation memory versus standard self-attention under explicit attention-matrix storage.
Abstract (from arXiv)
The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.
Related papers
- TAM: Task-Aware Memory Distillation for Efficient Spatiotemporal Prediction
- Glance Forcing: Making Bidirectional Diffusion Autoregressive at a Glance
- HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
- MeteoVerse: Unified Weather-Controllable Video World Model
- Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-08 UTC
Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling
Preprint posted: 2026-10-06 · Original record ↗
- arxiv
- 2610.09090
Explicit source versions (1)
- v1 · 2026-10-06
- Preprint available
Source date: 2026-10-06 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.
