Latent Dynamics · 2026-11-13
Latent Imagination for Parameterized Action Spaces
Atahan Cilan, \.Ihsan Mert Muhac{\i}ro\u{g}lu, \"Umit Can Bekar
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
HyLI is a model-based PAMDP agent that learns a hybrid mode-and-parameter policy entirely in the latent imagination of a DreamerV3-style world model and acts with a single forward pass. It uses counterfactual mode evaluation (CME) and success-window replay (SWR) to address PAMDP-specific learning issues. On seven PAMDP benchmarks, it matches or nears the strongest model-free baseline and is much faster than the planner DLPA.
Why it matters
Parameterized-action MDPs require choosing a discrete mode and continuous parameters that depend on the mode. HyLI targets the common trade-off in model-based PAMDP methods: sample efficiency versus slow action planning at decision time. By training in latent imagination and using CME/SWR, the authors aim to keep sample efficiency while making decisions fast enough for real-time control.
Method
- Trains a hybrid actor (discrete mode head + per-mode parameter heads) on imagined trajectories from a DreamerV3 RSSM, acting with one forward pass at deployment.
- Uses counterfactual mode evaluation (CME) to evaluate every (or top-probability) candidate mode with one imagined step per candidate from each imagined state, training the mode head via expected policy gradient and each parameter head with its own value.
- Uses success-window replay (SWR): draws a fraction of each batch from windows containing a success event to prevent rare early successes from being diluted.
Limitation
The candidate set for CME is all modes when K ≤ 16, otherwise it uses the 8 most probable plus 8 uniformly sampled modes with renormalized probabilities, described as a biased approximation whose cost does not grow with K.
Abstract (from arXiv)
Parameterized-action MDPs (PAMDPs) ask an agent to choose a discrete action together with continuous parameters whose meaning depends on that choice. Model-free PAMDP methods act fast but need many interactions; model-based methods are sample-efficient but plan at every decision, which makes them slow to act. We introduce HyLI (Hybrid Latent Imagination), a model-based agent that learns a hybrid policy entirely in the latent imagination of a world model and acts with a single forward pass. To address learning problems that the PAMDP structure creates, HyLI adds counterfactual mode evaluation, which gives every discrete action a learning signal at every imagined state, and success-window replay, which keeps rare successes in training. On seven standard PAMDP benchmarks, HyLI is within one published standard deviation of, or above, the published score of the strongest model-free baseline in our comparison on all 7 tasks, reaches 90% of that score with 2.4x fewer environment steps than that baseline (5.0x when its reward-free pre-training is counted), and decides 67x faster than the model-based planner DLPA at its published planner settings. It is the only method in our comparison that is both more sample-efficient than that baseline and fast enough for real-time control.
Related papers
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC
Latent Imagination for Parameterized Action Spaces
First announced by arXiv: 2026-11-13 · Original record ↗
- arxiv
- 2611.15744
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.15744v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Parameterized-action MDPs (PAMDPs) ask an agent to choose a discrete action together with continuous parameters whose meaning depends on that choice. Model-free PAMDP methods act fast but need many interactions; model-based methods are sample-efficient but plan at every decision, which makes them slow to act. We introduce HyLI (Hybrid Latent Imagination), a model-based agent that learns a hybrid policy entirely in the latent imagination of a world model and acts with a single forward pass. To address learning problems that the PAMDP structure creates, HyLI adds counterfactual mode evaluation, which gives every discrete action a learning signal at every imagined state, and success-window replay, which keeps rare successes in training. On seven standard PAMDP benchmarks, HyLI is within one published standard deviation of, or above, the published score of the strongest model-free baseline in our comparison on all 7 tasks, reaches 90% of that score with 2.4x fewer environment steps than that baseline (5.0x when its reward-free pre-training is counted), and decides 67x faster than the model-based planner DLPA at its published planner settings. It is the only method in our comparison that is both more sample-efficient than that baseline and fast enough for real-time control.
