Video World Models · 2026-10-08

Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling

Ryan Solgi, Parsa Madinei, Zheng Zhang

arXiv:2610.09090PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Tucker bottleneck attention (TuBA) replaces quadratic self-attention over multidimensional token grids with attention over a compact low-rank Tucker core. TuBA reads a hidden tensor into Tucker cores using learned mode-wise factors, applies multi-head self-attention on the cores, then writes the update back to the original space. An autoregressive variant uses block-causal masking on core slots. The authors evaluate on video prediction and global weather forecasting.

Why it matters

The paper targets a scalability limitation of self-attention for long multidimensional sequences by using a low-rank tensor structure as an inductive bias for global token mixing. The authors report accuracy–efficiency improvements versus standard self-attention and other efficient attention methods on video and weather forecasting, along with stated subquadratic compute from Tucker-core attention.

Method

  • TuBA treats hidden representations as multidimensional tensors, reads them into compact Tucker cores via learned mode-wise factors, applies standard multi-head self-attention on the core slots, then writes the update back to the ambient tensor space.
  • Autoregressive TuBA imposes causality at the sequence-of-tensors level using block-causal masking on concatenated core slots: within a tensor, core slots may interact; each tensor attends only to itself and earlier tensors.
  • Computational trade-off: Tucker core attention cost O(R^2 d + R d^2) plus mode-wise read/write contractions, yielding stated subquadratic complexity when R=o(N), with stated reduced FLOPs and activation memory versus standard self-attention under explicit attention-matrix storage.
Abstract (from arXiv)

The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-08 UTC

preprint

Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling

Ryan Solgi, Parsa Madinei, Zheng Zhang

Preprint posted: 2026-10-06 · Original record ↗

arxiv
2610.09090
Explicit source versions (1)
  • v1 · 2026-10-06
  1. Preprint available

    Source date: 2026-10-06 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.