Latent Dynamics · 2026-10-22
ALPS: Visual State Encoders for World Models
Yanqing Liu, Letian Zhang, Chengzhi Liu, Zijun Wang, Jieru Mei, Hongliang Fei, Yuyin Zhou, Cihang Xie
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
ALPS is a pretrained ViT-B/16 visual state encoder that unifies physical-state information, native language alignment, and pixel reconstruction in a single set of patch tokens. It is pretrained from scratch using language, reconstruction, and dense spatial correspondence objectives on still images and text, then refined via cross-view self-distillation. Frozen ALPS tokens support low LIBERO state-probe error and strong LIBERO policy success, and enable decoding predicted tokens into images for inspection.
Why it matters
The authors argue a single visual foundation should support prediction, instruction, and inspection for world models. ALPS targets a single frozen token interface (plus a native text tower) that simultaneously supports state probing, language-conditioned control, and pixel-renderable imagined futures, evaluated under a unified protocol across multiple public encoders.
Method
- Pretrain a ViT-B/16 image encoder from scratch on still images and text with four objectives on shared patch tokens: contrastive alignment, captioning, pixel reconstruction, and dense correspondence.
- Refine via cross-view self-distillation: a student predicts standardized patch tokens from a frozen Stage 1 teacher across crops/scale/appearance using a geometry-conditioned transformer predictor; then discard the predictor to form ALPS.
- Use frozen ALPS patch tokens for downstream world-model components (state probes, policies, dynamics predictors) and decode predicted future tokens into pixels via an evaluation decoder for visual inspection.
Limitation
The authors state that their cross-view refinement in Stage 2 uses only single-image supervision (still images and text alone, without video or interaction supervision), and they note that extending cross-view refinement to video clips and interaction trajectories is a natural next step.
Abstract (from arXiv)
World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models.
Related papers
- WorldCast: Exploring Intelligent Visual Generation in World State Prediction
- Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning
- What Matters for World Modeling in Vision-Language-Action Models?
- WOVEN: Weaving Visual World Modeling into Multimodal LLMs
- PrismWAM: Decoupling State Representations in World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-22 UTC
ALPS: Visual State Encoders for World Models
Preprint posted: 2026-10-21 · Original record ↗
- arxiv
- 2610.30120
Explicit source versions (1)
- v1 · 2026-10-21
- Preprint available
Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models.