Video World Models · 2026-10-14
ST-GRU PredRNN-V2: A GRU-Based Reformulation of PredRNN-V2 for Video Prediction
Augusta van Haren, Hanna Hoogen, Luca Pattavina, Quincy Rommen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose ST-GRU as a GRU-based replacement for PredRNN-V2’s ST-LSTM cells, using two GRU-style gates (update/reset) and merging the temporal memory state into the hidden state. They implement ST-GRU-based PredRNN-V2 (22.6M params) and evaluate on Moving MNIST using the original protocol, reporting MSE 55.1, SSIM 0.883, LPIPS 0.065, PSNR 19.53 dB averaged over ten frames.
Why it matters
This work tests whether simplifying the recurrent gating of ST-LSTM to GRU-style gates and reducing memory-state complexity can retain predictive performance in spatiotemporal video prediction, while also reducing parameters. The authors compare directly to ST-LSTM-based PredRNN-V2 and earlier baselines on Moving MNIST, including perceptual LPIPS and pixel-level metrics.
Method
- Develop a spatiotemporal GRU (ST-GRU) unit: replace ST-LSTM’s three gates per memory state with two GRU-style gates (update/reset) and omit the separate temporal memory state, integrating its function into the hidden state.
- Build ST-GRU-based PredRNN-V2 by replacing ST-LSTM blocks with ST-GRUs; because ST-GRU does not use the temporal memory state, train without PredRNN-V2’s decoupling loss and keep reverse scheduled sampling unchanged.
- Evaluate on Moving MNIST with the same training/testing protocol as Wang et al. (2021), reporting MSE, SSIM, LPIPS, and PSNR averaged over ten predicted frames; compare against multiple recurrent baselines.
Limitation
The authors evaluated only on Moving MNIST (a synthetic benchmark); they also report results from a single training run, and baseline results are taken from the literature rather than re-trained under identical conditions. They did not measure training time, inference time, or memory use, so practical computational benefits remain unquantified.
Abstract (from arXiv)
Video prediction requires models to capture complex spatiotemporal dependencies and to retain relevant information over long sequences. PredRNN and its successor PredRNN-V2 address this challenge with spatiotemporal Long Short-Term Memory (ST-LSTM) units that maintain two separate memory states, at the cost of a relatively complex recurrent building block. In this work, we investigate whether the simpler gating of Gated Recurrent Units (GRUs) can be transferred to spatiotemporal predictive learning. To this end, we develop a spatiotemporal GRU (ST-GRU) unit as a GRU-based alternative to the ST-LSTM. The ST-GRU replaces the ST-LSTM's three gates per memory state with two GRU-style gates (update and reset), and merges the temporal memory state into the hidden state, which reduces the number of parameters by about 5%. We incorporate the ST-GRU into PredRNN-V2 and evaluate the resulting model on the Moving MNIST video prediction benchmark, following the original experimental protocol. Averaged over ten predicted frames, the ST-GRU-based PredRNN-V2 achieves an MSE of 55.1, an SSIM of 0.883, an LPIPS of 0.065 and a PSNR of 19.53dB. It does not surpass the original ST-LSTM-based PredRNN-V2 on MSE, SSIM or PSNR, but it achieves a lower LPIPS and performs on par with the earlier ST-LSTM-based PredRNN, while using a simpler recurrent cell. These results suggest that ST-GRUs are a promising building block for spatiotemporal predictive learning and motivate further work on their design and integration.
Related papers
- TAM: Task-Aware Memory Distillation for Efficient Spatiotemporal Prediction
- World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models
- LOCI: Spatial Linear Memory for Streaming World Models
- Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning
- LongLive-Plug: Once-for-All Distillation for Video Generation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-14 UTC
ST-GRU PredRNN-V2: A GRU-Based Reformulation of PredRNN-V2 for Video Prediction
Preprint posted: 2026-10-13 · Original record ↗
- arxiv
- 2610.18645
Explicit source versions (1)
- v1 · 2026-10-13
- Preprint available
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Video prediction requires models to capture complex spatiotemporal dependencies and to retain relevant information over long sequences. PredRNN and its successor PredRNN-V2 address this challenge with spatiotemporal Long Short-Term Memory (ST-LSTM) units that maintain two separate memory states, at the cost of a relatively complex recurrent building block. In this work, we investigate whether the simpler gating of Gated Recurrent Units (GRUs) can be transferred to spatiotemporal predictive learning. To this end, we develop a spatiotemporal GRU (ST-GRU) unit as a GRU-based alternative to the ST-LSTM. The ST-GRU replaces the ST-LSTM's three gates per memory state with two GRU-style gates (update and reset), and merges the temporal memory state into the hidden state, which reduces the number of parameters by about 5%. We incorporate the ST-GRU into PredRNN-V2 and evaluate the resulting model on the Moving MNIST video prediction benchmark, following the original experimental protocol. Averaged over ten predicted frames, the ST-GRU-based PredRNN-V2 achieves an MSE of 55.1, an SSIM of 0.883, an LPIPS of 0.065 and a PSNR of 19.53dB. It does not surpass the original ST-LSTM-based PredRNN-V2 on MSE, SSIM or PSNR, but it achieves a lower LPIPS and performs on par with the earlier ST-LSTM-based PredRNN, while using a simpler recurrent cell. These results suggest that ST-GRUs are a promising building block for spatiotemporal predictive learning and motivate further work on their design and integration.