Latent Dynamics · 2026-10-21

LaLa-World: Scaling Latent World Models with Natural Language Actions

Junha Song, Geonmo Gu, Jeongeun Park, Heejeong Nam, Lucas Maes, Byeongho Heo, Dongyoon Han, Sangdoo Yun

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

LaLa-World is a language-conditioned latent world model that uses language as a shared interface for numerical actions from simulation/robotics and visual changes described in language from image/video data. Built with a language-model-based predictor and an MPC imagination planner, it reports higher average MPC planning performance than existing LWMs and comparable performance to an RAEv2 generative baseline with less training and faster planning.

Why it matters

The authors aim to scale latent world models across heterogeneous domains and data sources where action annotations differ in dimensionality/meaning and where image/video data lacks numerical action labels. By converting numerical actions into language and building the predictor on a language model, LaLa-World enables joint prediction across domains and supports planning with natural language action phrases, then provides pretrained initialization for downstream policy fine-tuning.

Method

Convert numerical simulation/robotics actions into language templates so diverse action spaces can be pooled to train a single predictor.
Train one latent transition predictor that takes encoded observations plus a language condition and predicts the next latent representation using squared error.
Use model predictive control (CEM over action sequences) with imagined latent rollouts conditioned by verbalized language actions; fine-tune the pretrained model into LaLa-Policy for joint action and future representation prediction.

Limitation

The authors note that their decoder “is not used during world model training,” and they describe that decoded images from LaLa-World may have lower visual quality because it is “not trained for pixel reconstruction.”

Abstract (from arXiv)

This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this end, we introduce LaLa-World, a Language-conditioned Latent World model that learns from simulation and robot trajectories together with broader image and video data. Two designs make this work. First, we describe both numerical actions and visual changes in language, enabling joint training of a single future predictor across heterogeneous datasets. Second, we build the predictor on a language model, allowing it to interpret language conditions while predicting future latent states. The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance and robustness to action phrasing. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with $6\times$ less training cost and $13.9\times$ faster planning. Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to $5.5\times$ faster inference.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC

preprint