Latent Dynamics · 2026-10-16
Learning Intrinsic Dynamics in Latent World Models
Jiamin Shi, Yuanhong Zhang, Zhiyang Liu, Jiawei Sun, Jingwen Fu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study how to preserve control-relevant “intrinsic dynamics” (action→consequence) under visual diversity. They propose that action predictability—reconstructing the executed action from consecutive latent states—discourages encoders from merging outcomes of different actions. They instantiate this with Concat IDM, which adds an inverse-dynamics loss during training for latent planning. On Push-T with 24 appearances, Concat IDM raises planning success from 9.1% to 77.2%.
Why it matters
Action-conditioned latent world models must keep distinctions between actions’ consequences even when observations vary for unrelated visual reasons. This paper links an inverse-dynamics-style action predictability objective to preserving those distinctions, with both theoretical penalties for outcome merging and empirical improvements in closed-loop latent planning under visual diversity.
Method
- Define action predictability as reconstructing the executed action from consecutive latent states, and hypothesize it preserves distinctions between action consequences under appearance variation.
- Derive a theoretical penalty (positive lower bound on squared inverse prediction error) when the encoder merges distinguishable outcomes of different actions into the same latent state; forward prediction can remain exact in that case.
- Instantiate the idea as Concat IDM: predict actions from concatenated endpoint latents [z_t; z_{t+1}], using action labels from offline trajectories to update the shared encoder while keeping the test-time planner unchanged.
Limitation
The principal finding concerns planning with seen appearances and matching current and goal renderings; success on a disjoint test palette is 4.2% for Plain JEPA and 13.4% for Concat IDM, leaving substantial room for transfer to unseen appearances.
Abstract (from arXiv)
Learning intrinsic dynamics---the relationship between actions and their consequences---is essential for world models. However, visual variations unrelated to the underlying control process can obscure this relationship in learned representations. We investigate the connection between action predictability, defined as the ability to reconstruct an action from consecutive latent states, and the preservation of intrinsic dynamics. Our central hypothesis is that encouraging action predictability discourages representations from discarding the distinctions between different actions and their consequences. We provide two complementary forms of evidence for this connection. Theoretically, we derive a positive lower bound on squared inverse prediction error when the encoder merges distinguishable outcomes of different actions from the same state and appearance, and show that forward prediction can still be exact on those transitions. Empirically, controlled appearance-diversity comparisons, supervision interventions, and latent probes support the role of action prediction in preserving control-relevant transition information. We instantiate action predictability with Concat IDM, which predicts actions from concatenated latent states and updates the shared encoder using action labels already available in offline trajectories, without changing the test-time planner. Across three independent training runs on Push-T with 24 appearances, Concat IDM increases planning success from 9.1% to 77.2%; its paired improvement is 52.9 percentage points larger than that observed with a single appearance. Together, these findings support action predictability as an inductive bias toward intrinsic dynamics and demonstrate the practical benefits of Concat IDM for latent planning under visual diversity.
Related papers
- Beyond Action Prediction: Learning Search-Free Planning in Latent Space
- Action-Consequence Alignment for Reliable Planning and Self-Improving in Latent World Models
- OneLA: Learning Many Worlds from Latent Actions
- Structured Latent Dynamics for Long-Horizon Embodied World Models
- Beyond One-Step Accuracy: State-Affine Latent Transition for Reliable Visual Planning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-15 · Indexed by Pub-AI: 2026-10-16 UTC
Learning Intrinsic Dynamics in Latent World Models
Preprint posted: 2026-10-15 · Original record ↗
- arxiv
- 2610.21612
Explicit source versions (1)
- v1 · 2026-10-15
- Preprint available
Source date: 2026-10-15 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Learning intrinsic dynamics---the relationship between actions and their consequences---is essential for world models. However, visual variations unrelated to the underlying control process can obscure this relationship in learned representations. We investigate the connection between action predictability, defined as the ability to reconstruct an action from consecutive latent states, and the preservation of intrinsic dynamics. Our central hypothesis is that encouraging action predictability discourages representations from discarding the distinctions between different actions and their consequences. We provide two complementary forms of evidence for this connection. Theoretically, we derive a positive lower bound on squared inverse prediction error when the encoder merges distinguishable outcomes of different actions from the same state and appearance, and show that forward prediction can still be exact on those transitions. Empirically, controlled appearance-diversity comparisons, supervision interventions, and latent probes support the role of action prediction in preserving control-relevant transition information. We instantiate action predictability with Concat IDM, which predicts actions from concatenated latent states and updates the shared encoder using action labels already available in offline trajectories, without changing the test-time planner. Across three independent training runs on Push-T with 24 appearances, Concat IDM increases planning success from 9.1% to 77.2%; its paired improvement is 52.9 percentage points larger than that observed with a single appearance. Together, these findings support action predictability as an inductive bias toward intrinsic dynamics and demonstrate the practical benefits of Concat IDM for latent planning under visual diversity.