Latent Dynamics · 2026-10-05
Learning Transferable Policies from Action-free Time Series Through Dynamical Embeddings
Niklas Emonds, Georgia Koppe
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose a hierarchical model-based RL framework for learning transferable control policies from action-free time series. A hierarchical dynamical system reconstruction (ALRNN/DSR) learns shared dynamics and per-system embeddings from uncontrolled recordings. These embeddings condition shared SAC policy/value networks, while interventions are modeled as additive latent perturbations with decoder-based constraints for interpretability and modality-protected control.
Why it matters
Action-free recordings lack observed intervention effects, making it hard to learn control without policies exploiting model errors or without knowing actuator coupling. The proposed approach constrains control via hierarchical parameter sharing, an explicit intervention model in latent space, and decoder-based constraints that make immediate intervention effects interpretable in observation space and enable transfer to related systems and embeddings-only adaptation.
Method
- Learn hierarchical dynamical system reconstruction (DSR) with an ALRNN piecewise-linear latent dynamics and subject embeddings; then freeze reconstruction parameters after training.
- Define actions as additive latent perturbations applied before the learned transition, restricted to latent dimensions that are decoded; train SAC entirely in simulation using the explicit intervention model.
- Interpret control through mechanistic analysis of ALRNN linear subregions (fixed points and stability), and adapt to unseen systems by fitting only a new subject embedding with fixed shared policy/value parameters.
Limitation
The authors state that due to the chaotic nature and heterogeneity of the dynamics across subjects and energy levels, learning a world model for the double pendulum is challenging and the prediction quality is limited.
Abstract (from arXiv)
Learning control from action-free recordings is challenging because intervention effects are unobserved and policies may exploit errors in reconstructed dynamics. We present a hierarchical model-based reinforcement learning framework that uses shared structure across related systems to learn system-specific control policies from action-free recordings. A hierarchical dynamical system reconstruction model captures shared dynamics and individual variation through low-dimensional embeddings. These embeddings are then reused to parameterize shared policy and value networks, linking differences in reconstructed dynamics to differences in control. Policies are trained entirely via simulation under an explicit intervention model with additive latent perturbations. Piecewise-linear recurrent neural networks enable mechanistic analyses of the controlled dynamics, while decoder-based constraints make the immediate effects of interventions interpretable in observation space and permit interventions on one modality while protecting another from direct manipulation. On Lorenz-63 and double-pendulum systems, hierarchical policies improve transfer over independently trained policies. On Lorenz-63, they also achieve a higher mean reward than repeated planning with the same reconstructed models, perform comparably to methods trained with controlled interactions, and generalize to systems absent from policy training after embedding inference alone. Applications to neural-behavioral recordings demonstrate suppression of predicted movement under constrained neural perturbations. Together, these findings show how shared dynamical representations support transferable control and mechanistic hypothesis generation from action-free recordings.
Related papers
- Higher-Order Action Supervision Makes A Strong Policy Class
- Identifiable World Models from Pretrained Diffusion Representations
- Residual Modeling Closes the Regression and Generative Policy Gap in Robot Learning
- Learning Intrinsic Dynamics in Latent World Models
- CausalDreamer: Learning Predictive World Models with Latent Disentanglement
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-05 UTC
Learning Transferable Policies from Action-free Time Series Through Dynamical Embeddings
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.03065
Explicit source versions (1)
- v1 · 2026-10-02
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Learning control from action-free recordings is challenging because intervention effects are unobserved and policies may exploit errors in reconstructed dynamics. We present a hierarchical model-based reinforcement learning framework that uses shared structure across related systems to learn system-specific control policies from action-free recordings. A hierarchical dynamical system reconstruction model captures shared dynamics and individual variation through low-dimensional embeddings. These embeddings are then reused to parameterize shared policy and value networks, linking differences in reconstructed dynamics to differences in control. Policies are trained entirely via simulation under an explicit intervention model with additive latent perturbations. Piecewise-linear recurrent neural networks enable mechanistic analyses of the controlled dynamics, while decoder-based constraints make the immediate effects of interventions interpretable in observation space and permit interventions on one modality while protecting another from direct manipulation. On Lorenz-63 and double-pendulum systems, hierarchical policies improve transfer over independently trained policies. On Lorenz-63, they also achieve a higher mean reward than repeated planning with the same reconstructed models, perform comparably to methods trained with controlled interactions, and generalize to systems absent from policy training after embedding inference alone. Applications to neural-behavioral recordings demonstrate suppression of predicted movement under constrained neural perturbations. Together, these findings show how shared dynamical representations support transferable control and mechanistic hypothesis generation from action-free recordings.