Latent Dynamics · 2026-11-10
K-PWM: Control-Oriented Structured Bilinear World Models under Partial Observation
Santosh Mohan Rajkumar, Sriram Narayanan, Samuel E. Otto, Debdipta Goswami
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
K-PWM is a probabilistic, control-oriented structured bilinear world model for partially observed systems with vector-valued measurements. It couples Koopman-bilinear latent dynamics with a nonlinear decoder and performs probabilistic latent-state inference via extended Kalman filtering and smoothing, without a separately trained inference network. It trains with a generalized expectation conditional maximization (GECM) procedure and uses the model for uncertainty-aware MPPI control.
Why it matters
The authors target world modeling and downstream control when observations are partial and noisy, and when the system provides vector measurements rather than full state. By combining structured Koopman-bilinear latent dynamics with model-induced probabilistic inference and uncertainty-aware planning, the method aims to improve data- and compute-efficiency for prediction and real-time capable control.
Method
- Couples Koopman-bilinear latent dynamics with a nonlinear observation decoder; maintains a Gaussian belief over latent states.
- Performs probabilistic latent-state inference via extended Kalman filtering and smoothing, without a separately trained inference network; exact Gaussian belief propagation through latent transitions for fixed actions.
- Trains with GECM: closed-form updates for Koopman generators, noise covariances, and latent prior, plus gradient-based decoder learning; uses Koopman-consistent initialization and selects latent dimension from data.
Limitation
In the Menagerie comparison, the authors attempted to evaluate Otto et al. [64] but report that EM iterations failed to converge in the tested configuration, leading to non-finite generator estimates or degenerate covariance matrices; they omit this baseline from the reported Menagerie comparison.
Abstract (from arXiv)
World models learn internal simulators of environment dynamics for prediction, planning, and control, yet learning them from limited, noisy, and partial observations remains challenging. While much progress has focused on pixel-based observations, many physical systems instead provide vector-valued measurements that reveal only part of the underlying state. We introduce K-PWM, a probabilistic world model for prediction and control under partial observation. K-PWM couples Koopman-bilinear latent dynamics with a nonlinear observation decoder and performs probabilistic latent-state inference through the learned model via extended Kalman filtering and smoothing, without a separately trained inference network. Its bilinear structure enables exact propagation of the maintained Gaussian belief through the latent transition and efficient rollouts for uncertainty-aware model predictive path integral (MPPI) control. For training K-PWM, we present a generalized expectation conditional maximization (GECM) procedure with closed-form updates for the generators, noise covariances, and latent prior, complemented by gradient-based decoder learning. A Koopman-consistent initialization selects the latent dimension from data and initializes the action-dependent dynamics. With limited training data, K-PWM achieves lower long-horizon prediction error than the evaluated baselines on partially observed Gymnasium MuJoCo systems, while substantially reducing training time relative to the strongest prediction baseline. On high-dimensional MuJoCo Menagerie robots, it achieves competitive tracking accuracy with low per-step computation times, demonstrating potential for real-time deployment. These results support structured probabilistic dynamics as a data- and compute-efficient approach to world modeling and control under partial observation.
Related papers
- $\Sigma$-LeWM: Learning World Models via S-Information for Robot Control
- Bilinear World Models: Learning Representations with Structured Dynamics for Efficient Control
- Structured Latent Dynamics for Long-Horizon Embodied World Models
- world2vec: Mechanistic world models through latent dynamics learning
- LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC
K-PWM: Control-Oriented Structured Bilinear World Models under Partial Observation
First announced by arXiv: 2026-11-10 · Original record ↗
- arxiv
- 2611.11521
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.11521v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
World models learn internal simulators of environment dynamics for prediction, planning, and control, yet learning them from limited, noisy, and partial observations remains challenging. While much progress has focused on pixel-based observations, many physical systems instead provide vector-valued measurements that reveal only part of the underlying state. We introduce K-PWM, a probabilistic world model for prediction and control under partial observation. K-PWM couples Koopman-bilinear latent dynamics with a nonlinear observation decoder and performs probabilistic latent-state inference through the learned model via extended Kalman filtering and smoothing, without a separately trained inference network. Its bilinear structure enables exact propagation of the maintained Gaussian belief through the latent transition and efficient rollouts for uncertainty-aware model predictive path integral (MPPI) control. For training K-PWM, we present a generalized expectation conditional maximization (GECM) procedure with closed-form updates for the generators, noise covariances, and latent prior, complemented by gradient-based decoder learning. A Koopman-consistent initialization selects the latent dimension from data and initializes the action-dependent dynamics. With limited training data, K-PWM achieves lower long-horizon prediction error than the evaluated baselines on partially observed Gymnasium MuJoCo systems, while substantially reducing training time relative to the strongest prediction baseline. On high-dimensional MuJoCo Menagerie robots, it achieves competitive tracking accuracy with low per-step computation times, demonstrating potential for real-time deployment. These results support structured probabilistic dynamics as a data- and compute-efficient approach to world modeling and control under partial observation.
