Video World Models · 2026-10-02

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation

Mikhail Dereviannykh, Vikram Voleti, Simon Donne, Mallikarjun Byrasandra Ramalinga Reddy, Shimon Vainer, Mark Boss

arXiv:2610.00686PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

SemanTok is a flexible-length, coarse-to-fine video tokenizer for autoregressive (AR) video generation. It keeps a VideoFlexTok-style decodable prefix recipe but adds semantic supervision by feeding frozen DINOv2 features to the encoder and training lightweight heads to reconstruct them from each retained token prefix alone. The authors report that a 201M SemanTok AR model matches or beats VideoFlexTok at 3.4× smaller size on fidelity/semantic alignment, with improved robustness across noise levels and OOD classes.

Why it matters

Token prefix quality controls how efficiently an AR video model can settle on global semantics before spending compute on later details. By ensuring that early prefixes carry DINO-based semantics (instead of relying only on an early decoder REPA layer that can be partly satisfied by noised inputs), SemanTok aims to improve semantic alignment and generation fidelity for smaller AR models and shorter token budgets.

Method

  • Build on VideoFlexTok’s flexible-length coarse-to-fine tokenizer, decodable from any token prefix via nested dropout and a time-causal AR interface.
  • Feed frozen DINOv2 patch features (dense) and a DINO class token into the tokenizer encoder; add lightweight causal cross-attention heads that reconstruct DINO features from each retained token prefix.
  • Train both heads to predict DINO features using only the token prefix (heads never see the noised latent), while keeping the decoder REPA loss as in VideoFlexTok.

Limitation

At the smallest budgets, the tokenizers are on par. The pure-noise difference is at most 0.02 at k = 1, and with a cleaner latent VideoFlexTok can be ahead up to k = 2 (σ=0.5) and up to k = 8 (σ=0.25), by at most 0.026. The authors also note SemanTok with 1 token per frame struggles to serve every objective.

Abstract (from arXiv)

Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model $3.4\times$ its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-02 UTC

preprint

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation

Mikhail Dereviannykh, Vikram Voleti, Simon Donne, Mallikarjun Byrasandra Ramalinga Reddy, Shimon Vainer, Mark Boss

Preprint posted: 2026-09-30 · Original record ↗

arxiv
2610.00686
Explicit source versions (1)
  • v1 · 2026-09-30
  1. Preprint available

    Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model $3.4\times$ its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.