Driving · 2026-10-28

Vision-Language-Action or World-Action? Just Action Models

Yonghao Yu, Lang Huang, Runyi Li, Toshihiko Yamasaki

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose Just Action Model (JAM): train a pretrained visual backbone with an action expert using only an action loss plus a “foresight” loss that predicts frozen encodings of recorded future observations via a training-only head and anchor branch. No visual generation is used in training or inference. JAM outperforms matched FastWAM and ImageWAM on LIBERO/LIBERO-Plus and matches or slightly improves on RoboTwin; caching foresight targets reduces online training FLOPs per sample per update.

Why it matters

JAM separates three choices behind VLA vs WAM differences: backbone initialization, objective on future observations, and what inference-time machinery is retained. By showing that future supervision can be learned without visual generation (using frozen future encodings), the work narrows what matters for control performance and training cost, and applies across generator, self-supervised, and vision-language backbones.

Method

Define JAM: pretrained visual backbone + action expert; train with action loss plus foresight loss that predicts frozen encodings of recorded futures using a prediction head discarded after training.
Recorded futures enter only the frozen future-anchor branch during training; at deployment only the backbone + action expert remain, so no future frames are generated in training or inference.
Compare fixed policy architectures (replacing visual generation in FastWAM/ImageWAM with foresight) and then apply the same foresight objective across backbone pretraining families (generative, self-supervised, vision-language).

Abstract (from arXiv)

The vision-language-action (VLA) and world-action model (WAM) communities debate whether control policies should inherit pretrained knowledge from vision-language models or from visual generators. The debate, however, bundles three separable choices: backbone initialization, the objective applied to future observations, and whether the machinery for that future loss is kept at inference. A gap between a VLA and a WAM therefore cannot simply be attributed to any one of them. We show that decoupling these choices yields a common framework for both families, which we term Just Action Model (JAM): a pretrained visual backbone with an action expert, trained with an action loss and a foresight loss. The foresight loss trains current-observation features to predict frozen encodings of the recorded future through a prediction head discarded after training, so no visual generation is involved in either training or inference, and the backbone can be chosen freely. We first compare objectives within fixed policy architectures, then evaluate the foresight recipe across backbone families. The resulting JAM policies outperform their matched FastWAM and ImageWAM counterparts on LIBERO and LIBERO-Plus, and achieve similar or slightly higher success rates on RoboTwin. Compared with FastWAM, JAM improves the success rates from 97.0%/55.3%/91.2% to 98.2%/64.5%/91.4%, respectively. With cached foresight targets, JAM reduces estimated online training FLOPs per sample per update by 73% relative to FastWAM. Our results show that future supervision improves control without visual generation or additional inference machinery, and strong foresight performance is attainable with either generator-initialized or vision-language backbones. Project Page: https://y0uroy.github.io/JAM

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC

preprint