Driving · 2026-11-10
$\Sigma$-LeWM: Learning World Models via S-Information for Robot Control
Xinrui Zu, Michael Przystupa, Zhao Yang, Shujian Yu, Kevin Sebastian Luck
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Σ-LeWM extends LeWorldModel with two additional auxiliary latent losses: a backward dynamics predictor and an inverse dynamics action predictor. The authors train these auxiliary objectives during training using decoupled contrastive learning (DCL), and frame the overall method as maximizing S-Information across forward, backward, and inverse directions. Experiments on standard benchmarks and simulated robot manipulation show better performance and improved planning efficiency over LeWM.
Why it matters
Robot control with learned world models benefits from structured latent embeddings that support efficient planning and stable long-horizon rollouts. By adding backward and inverse latent dynamics losses (trained via contrastive objectives) on top of LeWM’s forward prediction and latent regularization, Σ-LeWM targets control-relevant structure and reduced test-time planning compute.
Method
- Start from decoder-less LeWorldModel with a latent transition f(z_t,ã_t) trained with a forward latent prediction loss plus SIGReg to prevent mode collapse.
- Add training-only auxiliary tasks: a backward dynamics predictor f_back(z_{t+1},ã_t) and an inverse dynamics predictor f_inv(z_t,z_{t+1}).
- Train backward and inverse objectives with decoupled contrastive learning (DCL), motivated as maximizing S-Information (a sum of multivariate dependence terms).
Limitation
The paper’s future work will investigate using the current auxiliary tasks (backward and inverse latent dynamics) at planning time and deploying Σ-LeWM on real-world robot tasks.
Abstract (from arXiv)
World models have increasingly been applied as a model-based control method for robotics tasks such as manipulation and locomotion. While early world model architectures relied on decoders to prevent mode collapse and learn a predictable latent space, recent works on world models, such as Joint Embedding Predictive Architecture or LeWorldModel, increasingly aim to learn embeddings without a grounding decoder, utilising only a latent forward dynamics predictor and a stabilising entropy term. In this paper, we set out to demonstrate that the further introduction of two additional latent losses to LeWorldModel, a backward dynamics predictor and an inverse dynamics action predictor, enforces further structure on the latent embedding, leading to further gains in both performance and planning efficiency. These are critical properties for improving the applicability of such methods to real-time model-based control of real-world robots. We demonstrate these gains on standard world model benchmarks and two simulated robot manipulation tasks.
Related papers
- Continuous-Action World Models: Learning Cross-Embodiment Spatiotemporal Dynamics Priors
- LeWAM: End-to-End World and Action Modeling with JEPAs
- Diffusion-Steering-Based MPC with a JEPA World Action Model
- Copper-Policy: Focus on the Representation for Robust Robot Manipulation
- LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-09 · Indexed by Pub-AI: 2026-11-10 UTC
$Σ$-LeWM: Learning World Models via S-Information for Robot Control
Preprint posted: 2026-11-09 · Original record ↗
- arxiv
- 2611.12732
Explicit source versions (1)
- v1 · 2026-11-09
- Preprint available
Source date: 2026-11-09 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.12732v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World models have increasingly been applied as a model-based control method for robotics tasks such as manipulation and locomotion. While early world model architectures relied on decoders to prevent mode collapse and learn a predictable latent space, recent works on world models, such as Joint Embedding Predictive Architecture or LeWorldModel, increasingly aim to learn embeddings without a grounding decoder, utilising only a latent forward dynamics predictor and a stabilising entropy term. In this paper, we set out to demonstrate that the further introduction of two additional latent losses to LeWorldModel, a backward dynamics predictor and an inverse dynamics action predictor, enforces further structure on the latent embedding, leading to further gains in both performance and planning efficiency. These are critical properties for improving the applicability of such methods to real-time model-based control of real-world robots. We demonstrate these gains on standard world model benchmarks and two simulated robot manipulation tasks.
