Latent Dynamics

Dreamer/RSSM-style latent dynamics and model-based RL. (242 papers)

2026-11-13 · arXiv:2611.14880

FOE-Dreamer: Deployment-Efficient Learning of Cyber Defense Policies in Operational Networks

Yinuo Du, Christopher Kiekintveld

FOE-Dreamer is a model-based RL framework for cyber defense that trains online and directly in an emulated operational network on cloud infrastructure, without a simulator stage. It uses a factored world model separating defender-driven, exogenous user activity, and opponent dynamics inferred from defender-observable signals. On an 8-host OpenStack testbed with GHOSTS user emulation and two scripted attacker profiles, it trains in 3 days on a single GPU and reduces episode loss by about 2× vs Rainbow and IQN under matched compute.

Latent DynamicsAI summary
2026-11-13 · arXiv:2611.15048

Does Your World Model Know Actions Have Consequences?

Ayush Gaggar, Jared Berry, Todd D. Murphey

The authors argue current embodied world-model evaluation misses how well action-conditioned dynamics predict counterfactual consequences. They introduce an Action Perturbation (AP) sensitivity metric comparing predicted vs true impulse responses under forced action perturbations. Using frozen TD-MPC2 components and finetuned dynamics heads trained with different data-collection regimes, they show noise-injected heads improve action sensitivity while workspace raster performs worst.

Latent DynamicsAI summary
2026-11-13 · arXiv:2611.15744

Latent Imagination for Parameterized Action Spaces

Atahan Cilan, \.Ihsan Mert Muhac{\i}ro\u{g}lu, \"Umit Can Bekar

HyLI is a model-based PAMDP agent that learns a hybrid mode-and-parameter policy entirely in the latent imagination of a DreamerV3-style world model and acts with a single forward pass. It uses counterfactual mode evaluation (CME) and success-window replay (SWR) to address PAMDP-specific learning issues. On seven PAMDP benchmarks, it matches or nears the strongest model-free baseline and is much faster than the planner DLPA.

Latent DynamicsAI summary
2026-11-13 · arXiv:2611.16375

Recursive State-Augmented Language Models: Persistent Latent-State Adaptation of Pretrained Language Models

Omer Haq

Recursive State-Augmented Language Models (Recursive SALM) maintain a persistent latent state that evolves after each observation and generates adaptations of a frozen pretrained language model via a modular read–transition–write architecture. In deterministic experiments using frozen GPT-2 small, latent-gated LoRA matches a full-context LoRA baseline on a four-turn hypothesis-inference task. Latent-state interventions show predictions depend on the state when earlier evidence is absent from the decoder context.

Latent DynamicsAI summary
2026-11-13 · arXiv:2611.16534

Context-Predictive Low-Rank Adaptation for Face Presentation Attack Detection

Aminu Yusuf, Shakeel Muhammad Ibrahim, Sulaimon Oyeniyi Adebayo et al.

CP-JEPA adapts a pretrained CLIP (ViT-B/16) visual encoder for face presentation attack detection using low-rank adaptation (LoRA) plus context-predictive learning. A student predicts complete-image teacher features at masked spatial locations (EMA teacher targets) while retaining unmasked classification, masked classification, and prediction consistency. Inference uses only the LoRA-adapted CLIP visual encoder and binary classifier.

Latent DynamicsAI summary
2026-11-13 · arXiv:2611.16645

Rethinking Actions: Continuity-Aware Priors for World-Model Planning

Shaobo Yang, Zichen Ai, Ziyang Ma et al.

The authors propose LG-DKGP, a locality-gated deep-kernel Gaussian-process prior for visual world-model planning that learns action continuations from the temporal structure of expert demonstrations. Their planner uses a learned neural mean plus a temporal GP residual to form a continuity-aware prior, initializes CEM search with a training-derived exploration scale, and evaluates progress with a frozen world model.

Latent DynamicsCode availableAI summary
2026-11-13 · arXiv:2611.16647

InertiaMPDD: Multimodal Personalized Depression Detection via Emotional Inertia-Inspired Dynamics Modeling

Changzeng Fu, Songyang Wang, Chang Gao et al.

InertiaMPDD learns a healthy next-state predictor from non-depressed recordings using frozen pretrained multimodal representations. It compares predicted vs observed next latent states to form reference-derived descriptors: reference deviation, residual decay rate (AR(1) on residuals), and a Jacobian spectral descriptor. These proxies support standalone screening and a personalized correction to a multimodal population margin, achieving high fusion F1 on D-Vlog and F1 on LMVD, with limited standalone discrimination on AVEC2014.

Latent DynamicsAI summary
2026-11-13 · arXiv:2611.17031

Your Vision Foundation Model Is Secretly an EEG Foundation Model

Guoan Wang, Feng Liu

SeeEEG converts multichannel EEG into images that a frozen vision transformer can read, by assigning each EEG channel to a dedicated row of ViT tokens and learning only the “display” (montage/filter/contrast) parameters plus a lightweight readout head. Using a frozen DINOv3 ViT-B backbone, the authors report higher balanced accuracy than seven EEG foundation models across eight datasets and strong mean rank improvements in Cohen kappa/AUROC.

Latent DynamicsAI summary
2026-11-13 · arXiv:2611.17110

SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models

Jiachun Jin, Zetong Zhou, Haonan Wang et al.

SALA trains a latent action tokenizer with semantic alignment across human and robot embodiments using unpaired atomic-action language descriptions. Language similarities form soft neighborhoods that move human latents toward control-grounded robot latents (robot latents anchored by action reconstruction). The aligned shared latent action space enables scaling: more general human-video pre-training reduces robot action prediction error; more task-specific human-video post-training increases real-robot success rates.

Latent DynamicsAI summary
2026-11-11 · arXiv:2611.13113

Building World Models from Intrinsic Motivation

Reece Keller, Declan Kutscher, Xaq Pitkow et al.

The authors integrate 24 intrinsic objectives (curiosity, entropy, competency) into a common DreamerV3 model-based RL framework, alongside three representation-learning strategies (observation reconstruction, next-embedding prediction, current-embedding redundancy reduction). They evaluate zero-shot, few-shot, and multi-task control across locomotion, manipulation, and open-ended gaming. Curiosity with observation reconstruction gives the strongest overall family profile; episodic+lifelong novelty yields the best compositional vision transfer and adaptation.

Latent DynamicsAI summary
2026-11-11 · arXiv:2611.13348

ProTEUS: A mutation-resolved world model for closed-loop protein evolution

Yikun Zhang, Minghao Guo, Guangxue Xu et al.

ProTEUS is a mutation-resolved world model for closed-loop protein evolution. Pretrained on deep mutational scanning, it learns a joint latent representation of sequence, evolutionary, and structural information with MutationFormer, where mutations are actions and predicted outcomes guide exploration. It also uses a fast physics-constrained all-atom predictor, and refines via closed-loop experimental feedback. The authors report improved mutation prediction and >10× activity over wild type in two engineering campaigns.

Latent DynamicsAI summary
2026-11-11 · arXiv:2611.13592

Few-Shot Residual Adaptation of Pretrained Latent World Models

Jakub Suliga, Pankhuri Vanjani, Nisarga Nilavadi et al.

The authors study few-shot residual adaptation for pretrained latent world models under physical dynamics shifts. They keep the pretrained DUET-DINO/DINOv3 components frozen and learn self-supervised residual corrections from latent prediction errors, applied both inside the latent predictor and to its predicted latent state. On a real robot with center-of-mass and friction shifts, frozen success drops near zero; with three adaptation episodes, dual-site recovers 90% (CoM) and 100% (friction).

Latent DynamicsAI summary
2026-11-11 · arXiv:2611.14419

Evaluating LLM-Generated World Models for Neuro-Symbolic Reasoning

Souhaila Djaffal, Amna Jamal, Gagana Madhugiri Devaraja et al.

The authors evaluate whether five LLMs can generate executable world models as partially observable stochastic games (POSGs) from natural-language scenarios. They introduce a 44-scenario benchmark and a three-lane evaluation (mechanical validity, reference-relative structural agreement, and semantic checks). In the reference-backed game track, 134/135 pass a strict mechanical audit, but mean permutation-invariant structural F1 is only 0.366, and just 28/134 exactly match the authored reference.

Latent DynamicsAI summary
2026-11-10 · arXiv:2611.10151

WorldMind: Test-Time Scaling for Visual Spatial Reasoning through Latent World Simulation

Haoming Wang, Yongjian Fu, Ju Ren et al.

WorldMind is a test-time scaling framework for visual spatial reasoning that performs iterative “latent world simulation.” A VLM selects spatial transformations; a world model executes them and returns aligned intermediate latent features directly to the VLM, avoiding repeated decode/re-encode through RGB. The authors train latent selection/alignment, then fine-tune with simulation-trajectory supervision and process-verifiable reinforcement learning using step-wise rewards.

Latent DynamicsAI summary
2026-11-10 · arXiv:2611.10251

TORNADO: Adaptive Latent Stochastic Transport for Calibrated Probabilistic PDE Forecasting

Jorge Mifsut-Benet, Armand Kassa\"i Koupa\"i, Ramon Daniel Regueiro-Espino et al.

TORNADO is a latent-space probabilistic PDE forecasting framework that models the next-step as stochastic transport between VAE encoder posteriors using a learned Itô SDE. Its key idea is state-dependent diffusion (learned drift and diffusion) to adapt predictive uncertainty to local latent dynamics. Training uses simulation-free analytic conditional endpoint targets from a Gaussian interpolant plus an energy-distance regularizer.

Latent DynamicsAI summary
2026-11-10 · arXiv:2611.10523

Action Transfer Across Character Body Plans in Latent-Action World Models: An Empirical Study of Animated Characters

Amarnath R

The authors study whether latent-action world models can transfer an action learned from one 2D animated character body plan to a different body plan so that the target performs the action in its own way. They introduce an action-specific transfer score and evaluate five training objectives on a synthetic 2D benchmark (six body plans), Sprites, and 431 DeformingThings4D animal/human animations. They find that fixed-camera “transfer” can be high even when the action is ignored; body-centered transfer reaches only 0.46 for unseen animals and 0.20 for humans.

Latent DynamicsAI summary
2026-11-10 · arXiv:2611.10660

Dynamics-Informed Action Learning for Goal-Conditioned Latent Planning

Yinuo Jiang, Guangzhi Wang, Yiran Wang et al.

DIAL (Dynamics-Informed Action Learning) trains latent dynamics with rollout alignment and trains a lightweight inverse dynamics head with multi-step inverse supervision. At test time, the inverse and forward dynamics are coupled to generate a goal-directed action sequence that initializes only the mean of CEM’s proposal distribution; standard CEM sampling/refinement then improves the plan. The authors report DIAL outperforms prior latent world-model and planning baselines on four visual control benchmarks.

Latent DynamicsAI summary
2026-11-10 · arXiv:2611.10771

WorldFoundry: Towards Unified Infrastructure for World Intelligence

Juanxi Tian, Jiadong Hou, Baiang Li et al.

WorldFoundry proposes a functional definition of generative world models as history- and control-conditioned dynamical systems with persistent environmental state, action-/intervention-conditioned forward evolution, and closed-loop observational updates. It presents open-source infrastructure to unify model/benchmark discovery, reproducible inference, artifact management, evaluation, and visualization across video, 3D/4D, embodied action, and interactive worlds.

Latent DynamicsCode availableAI summary
2026-11-10 · arXiv:2611.11140

SurgWorld: Learning Surgical Representations Beyond Observation via Latent Prediction

Jingkun Yue, Pengxiang Wang, Siqi Zhang et al.

SurgWorld is a video-native, self-supervised framework for surgical representation learning beyond visible observations. It treats a relative spatio-temporal shift as a pseudo-action and uses Out-of-Sight Dynamics Modeling (OSDM) to predict the target latent representation from a partial context. A Geometric Visual Similarity (GVS) loss aligns patch-level relational structures with a frozen teacher. An agglomerative training framework integrates multiple teachers to learn unified latent dynamics and geometry.

Latent DynamicsAI summary
2026-11-10 · arXiv:2611.11242

DeepSoMo: Deformable Object Motion Estimation under Proprioceptive Partial Observability via Predictive Latent Dynamics

Hanwen Zhao, Weicheng Dai, Luoyao Chen et al.

DeepSoMo estimates deformable object motion under proprioceptive partial observability using latent temporal consistency. It encodes internally rendered deformation images, aggregates them with an LSTM latent dynamics model, and enforces a self-supervised latent prediction constraint so latent states remain predictive of future latent features. Evaluated in high-fidelity FEM simulation, it reports consistent improvements over architecture-matched recurrent baselines for long-horizon trajectory estimation and related tasks.

Latent DynamicsAI summary